누군가 열심히 개발한 알고리즘을 개발했다면, 내가 처음 코드를 만든다면 할만하다. 그러나 인터넷에 시간이 남아도는 인간이 많고 그들을 다 제치고 내가 처음이 아닌 확률이 상당히 크다. 누군가 만든 고급진 코드를 충분하게 찾아 볼 필요가 있고, 만약 있다면 다시 할 필요는 없다. 다시해도 그 성능을 넘어설 수 없다. 찾아보니 강화학습을 쉽게 사용할 수 있는 keras-rl을 찾았다. 2.0 버전에 맞도록 구현된 keras-rl2를 설치하면 된다.
python -m pip install keras-rl2
사용자는 각자에 필요한 환경을 설정하면 된다. 사용자가 이 부분에서 삽질해야 하고, 가치 있다. opanai-gym은 인공지능 알고리즘을 쉽게 개발하기 위해 설정한 환경이고, keras-rl은 누군가 개발한 알고리즘을 쉽게 사용하기 위한 방법이다. 각자 환경을 openai-gym 형식에 맞춰 넣으면 된다. 다음 tutorial을 보면 된다.
dqn, a2c 등 유명한 알고리즘을 구현했다. 내가 필요한 a3c을 구현할 때 까지 기다리면 된다.
튜토리얼을 돌리면 에러난다. display가 제대로 설정되지 않았다. 내가 필요한 환경에서는 굳이 display가 필요없다.
tf-docker /home/mnt/keras-rl > python test.py
2020-11-07 23:00:09.425948: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
flatten (Flatten) (None, 4) 0
_________________________________________________________________
dense (Dense) (None, 16) 80
_________________________________________________________________
activation (Activation) (None, 16) 0
_________________________________________________________________
dense_1 (Dense) (None, 16) 272
_________________________________________________________________
activation_1 (Activation) (None, 16) 0
_________________________________________________________________
dense_2 (Dense) (None, 16) 272
_________________________________________________________________
activation_2 (Activation) (None, 16) 0
_________________________________________________________________
dense_3 (Dense) (None, 2) 34
_________________________________________________________________
activation_3 (Activation) (None, 2) 0
=================================================================
Total params: 658
Trainable params: 658
Non-trainable params: 0
_________________________________________________________________
None
2020-11-07 23:00:10.496021: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-11-07 23:00:10.510795: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-07 23:00:10.511178: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:26:00.0 name: GeForce GTX 1060 6GB computeCapability: 6.1
coreClock: 1.7085GHz coreCount: 10 deviceMemorySize: 5.93GiB deviceMemoryBandwidth: 178.99GiB/s
2020-11-07 23:00:10.511203: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-11-07 23:00:10.512417: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-11-07 23:00:10.513603: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-11-07 23:00:10.513800: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-11-07 23:00:10.515074: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-11-07 23:00:10.515803: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-11-07 23:00:10.518627: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-11-07 23:00:10.518842: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-07 23:00:10.519243: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-07 23:00:10.519566: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-11-07 23:00:10.519919: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-11-07 23:00:10.542963: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3399500000 Hz
2020-11-07 23:00:10.543724: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x52a1590 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-11-07 23:00:10.543767: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-11-07 23:00:10.799878: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-07 23:00:10.800523: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4baece0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-11-07 23:00:10.800604: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1060 6GB, Compute Capability 6.1
2020-11-07 23:00:10.801142: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-07 23:00:10.802326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:26:00.0 name: GeForce GTX 1060 6GB computeCapability: 6.1
coreClock: 1.7085GHz coreCount: 10 deviceMemorySize: 5.93GiB deviceMemoryBandwidth: 178.99GiB/s
2020-11-07 23:00:10.802405: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-11-07 23:00:10.802461: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-11-07 23:00:10.802499: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-11-07 23:00:10.802542: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-11-07 23:00:10.802582: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-11-07 23:00:10.802620: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-11-07 23:00:10.802660: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-11-07 23:00:10.802864: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-07 23:00:10.803808: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-07 23:00:10.804611: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-11-07 23:00:10.804690: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-11-07 23:00:11.184178: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-07 23:00:11.184228: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
2020-11-07 23:00:11.184238: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
2020-11-07 23:00:11.184452: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-07 23:00:11.184839: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-07 23:00:11.185183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4853 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:26:00.0, compute capability: 6.1)
Training for 50000 steps ...
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training_v1.py:2070: Model.state_updates (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version.
Instructions for updating:
This property should not be used in TensorFlow 2.0, as updates are applied automatically.
2020-11-07 23:00:11.519149: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/gym/envs/classic_control/rendering.py", line 25, in <module>
from pyglet.gl import *
File "/usr/local/lib/python3.6/dist-packages/pyglet/gl/__init__.py", line 95, in <module>
from pyglet.gl.lib import GLException
File "/usr/local/lib/python3.6/dist-packages/pyglet/gl/lib.py", line 149, in <module>
from pyglet.gl.lib_glx import link_GL, link_GLU, link_GLX
File "/usr/local/lib/python3.6/dist-packages/pyglet/gl/lib_glx.py", line 45, in <module>
gl_lib = pyglet.lib.load_library('GL')
File "/usr/local/lib/python3.6/dist-packages/pyglet/lib.py", line 164, in load_library
raise ImportError('Library "%s" not found.' % names[0])
ImportError: Library "GL" not found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 45, in <module>
dqn.fit(env, nb_steps=50000, visualize=True, verbose=2)
File "/usr/local/lib/python3.6/dist-packages/rl/core.py", line 187, in fit
callbacks.on_action_end(action)
File "/usr/local/lib/python3.6/dist-packages/rl/callbacks.py", line 100, in on_action_end
callback.on_action_end(action, logs=logs)
File "/usr/local/lib/python3.6/dist-packages/rl/callbacks.py", line 362, in on_action_end
self.env.render(mode='human')
File "/usr/local/lib/python3.6/dist-packages/gym/core.py", line 240, in render
return self.env.render(mode, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/gym/envs/classic_control/cartpole.py", line 174, in render
from gym.envs.classic_control import rendering
File "/usr/local/lib/python3.6/dist-packages/gym/envs/classic_control/rendering.py", line 32, in <module>
''')
ImportError:
Error occurred while running `from pyglet.gl import *`
HINT: make sure you have OpenGL install. On Ubuntu, you can run 'apt-get install python-opengl'.
If you're running on a server, you may need a virtual frame buffer; something like this should work:
'xvfb-run -s "-screen 0 1400x900x24" python <your_script.py>'