How to solve goseek problem with DQN ?

[han-kyung-min]

Hello,

I modified the demo code: "goseek-ppo-ipynb" in order to use DQN instead of PPO2. The code seems running OK for a while. Then, it suddenly stops with the message, "The kernel appears to have died. It will restart automatically" Then, the agent stops navigating/learning.

Could you please give me an advice on the code snippet below?

================= What I modified is as below =====================

`policy_kwargs = {'cnn_extractor': image_and_pose_network}

class CustomDQNPolicy(FeedForwardPolicy):
def init(self, *args, **kwargs):
super(CustomDQNPolicy, self).init(*args, **kwargs,
layers=[32, 32],
layer_norm=False,
feature_extraction="mlp")

model = DQN(
CustomDQNPolicy,
env,
verbose=1,
tensorboard_log="./tensorboard/",
learning_rate=0.00025,
policy_kwargs=policy_kwargs,
)`

===========================================================

Thanks

[ZacRavichandran]

Hi,

Thanks for sharing! I'll look into your changes further. In the meantime, did you get any further output before the process died? It could be helpful for debugging.

[han-kyung-min]

Hello,

The attached picture shows the outputs at my terminal.
Hope this helps you for the debugging.

[ZacRavichandran]

Looking into this further, the stable-baselines DQN docs note that

The DQN model does not support stable_baselines.common.policies, as a result it must use its own policy models (see DQN Policies).

Perhaps the use of stable_baselines.common.policies.FeedForwardPolicy is causing the issue?

[han-kyung-min]

What I do is from stable_baselines.deepq.policies import FeedForwardPolicy. Importing FeedForwardPolicy from common.policies won't let me start the program. My code works fine for a while (about 5~10 mins). Then, it stops with "Kernel Restarting" message above.

Does your DQN run OK at your side ?

[ZacRavichandran]

I'm not very familiar with their DQN implementation, but the error could be coming from the feature extractor used. It looks like you're using feature_extraction="mlp". However, the observations consist of images and a pose vector which would not be compatible with a feed forward network. Perhaps changing the feature extractor to "cnn" and then passing in the custom cnn defined in the notebook via policy_kwargs will work?

On another note, sometimes the error messages given by SubprocVecEnv can be opaque, switching to DummyVecEnv could help for the purposes of debugging.