High Policy Loss in SAC_CQL

[waffoo]

policy_loss in SAC_CQL is significantly higher than the official implementation when tested with hopper-expert-v0 in d4rl.

accel/accel/agents/sac_cql.py

Line 261 in af3f511

policy_loss = ((self.alpha * log_pi) - min_qf_pi).mean()

With the author's implementation, we can get the loss lower than -350, while using accel we can't even reach -300, which leads to slower and unstable learning.

[waffoo]

The cause of unstable performance seems to be because I treat timeout frames as done frames in the preprocessing here.
I'll fix the line later, but the high policy loss issue still needs to be investigated.