8

I am trying to apply the PPO algorithm from the stable baselines3 library https://stable-baselines3.readthedocs.io/en/master/ to a custom environment I made.

One thing I don't understand is the following line:

mean_reward, std_reward = evaluate_policy(model, env, n_eval_episodes=10, deterministic=True)

Should I always let deterministic equal True? When I keep deterministic="True", my custom environment "somehow" is always solved (i.e., always returning reward of 1 +/- 0 std).

And when I change it to "False", it starts behaving in a reasonable way (i.e., sometimes it succeeds (reward=1) and sometimes it fails (reward=0).

11
  • 3
    If I am not mistaken, stable baselines takes a random sample based on some distribution when using deterministic is False. This means that if the model prediction is not sure of what to pick, you get a higher level of randomness, which increases the exploration. During evaluation you generally don't want to explore, but exploit the model. Therefore deterministic should be True, which always returns the best action. When using deterministic is False, you won't always get the best action, but sometimes less optimal action picked at random (based on your model confidence).
    – Thymen
    Mar 3, 2021 at 11:05
  • I actually tested it (deterministic= "True") before and after training the model. And even before training the model, the reward was always 1, which was very unusual. Can you explain why an untrained model would be successful!
    – mac179
    Mar 3, 2021 at 15:30
  • I don't know the environment you are training on, can you provide some action distributions of the model (with deterministic = True and False) and a small explanation of your environment? That would help thinking about what goes wrong. My thoughts so far is that the default model initialization already solves the environment, and you never get a different action (that makes you fail).
    – Thymen
    Mar 3, 2021 at 16:01
  • 1
    Running your code for 100_000 steps and Determinstic=True, leads to a start of 0. and end of 49. With Determinstic=False, start 0. and end 31. Which seem reasonable. For the rendering, the reason that it is slow is because you are re rendering the whole plot every time with more data. The best way to handle that is either making it a separate process and using a queue to transfer the data. Or make a render interval, every 20 steps for example.
    – Thymen
    Mar 5, 2021 at 10:32
  • 1
    An example of plotting using a separate process can be found here.
    – Thymen
    Mar 5, 2021 at 13:13

1 Answer 1

10

This parameter corresponds to "Whether to use deterministic or stochastic actions". So the thing is when you are selecting an action according to given state, the actor_network gives you a probability distribution. For example for two possible actions a1 and a2: [0.25, 0.75]. If you use deterministic=True, the result will be action a2 since it has more probability. In the case of deterministic=False, the result action will be selected with given probabilities [0.25, 0.75].

1
  • 1
    So, basically Deterministic Policy when deterministic=True and Stochastic Policy otherwise? Mar 14, 2022 at 6:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.