Hindsight PRIOR #12
Hindsight PRIOR #12
Conversation
There was a problem hiding this comment.
Hey Shengjun, would you please separate the changes of GPT2 and keep the implementation of GPT the same as before? I'm worried about the backward compatibility about this change since there are lots of algorithms depending on GPT2. You can unfold the code of GPT2 in TWM and add the interface you want (need_weights). Thanks!
There was a problem hiding this comment.
I made the changes with compatibility, and it will not affect the existing algorithms. GPTBlock is only used in GPT2. And GPT2's output is the same as the original output when output_attentions defaults to False
| reward_reg: 0.0 | ||
| max_seq_len: 100 | ||
| world_steps: 10000 | ||
| prior_coef: 1000 |
There was a problem hiding this comment.
This seems too huge, could you record the scale of rm_loss and prior_loss and see whether this term actually dominates the reward learning? By the way, what's the default value of prior_coef in authors' impl.?
There was a problem hiding this comment.
Origin paper: 1000 (MetaWorld), 5 (DMC)
Record:
| prior_coef | rm_loss | prior_loss |
|---|---|---|
| 1 | 0.3 | 0.015 |
| 100 | 0.5 | 1e-4 |
| 1000 | 0.7 | 5e-6 |
Hindsight PRIORs for Reward Learning from Human Preferences
和原版不同，这里使用了 GPT2 架构，并且没有 observation model，而是直接使用一个回归预测头，自回归地预测 next_obs。