why there are two calls to the policy, also where is the non intrinsic characteristic of intrinsic reward?

[mehdimashayekhi]

Hi, Thanks for sharing. I was wondering if you can explain why do we need two calls for apply_policy in the can_gru_policy_dynamics.py, here

random-network-distillation/policies/cnn_gru_policy_dynamics.py

Line 69 in f75c0f1

self.apply_policy(self.ph_ob[None][:,:-1],

and here

random-network-distillation/policies/cnn_gru_policy_dynamics.py

Line 83 in f75c0f1

self.apply_policy(self.ph_ob[None],

Also, I have another question. Based on the paper, intrinsic reward, should be non episodic but extrinsic reward is treated as episodic, I couldn't find where this "non episodic" charactersitic has been addressed for intrinsic reward in the implementation. Shouldn't we also add this episodic reward (i.e., eprews) to the external reward (i.e., rews_ext)?!

random-network-distillation/ppo_agent.py

Line 241 in f75c0f1

eprews = MPI.COMM_WORLD.allgather(np.mean(list(self.I.statlists["eprew"])))

really appreciate your responses

[harri-edwards]

There are two graphs created for the policy / predictor, one for rollout and one for optimization. This is because at rollout time the time dimension has size 1 and is better treated separately.

If you look at

random-network-distillation/ppo_agent.py

Line 294 in f75c0f1

self.I.buf_advs = self.int_coeff*self.I.buf_advs_int + self.ext_coeff*self.I.buf_advs_ext

you'll see the intrinsic and extrinsic advantages are combined there.