[algo] Adding CISPO policy loss

[twkillian]

What does this PR do?

This PR adds CISPO to core_algos.py to start integrations toward a full implementation of scaleRL.

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here: ...
• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
  • {modules} include fsdp, megatron, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data
  • If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
  • {type} is in feat, fix, refactor, chore, test
  • If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
  • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

API and Usage Example

CISPO is a sampled policy gradient loss that adopts a lot from the REINFORCE family of algorithms. How it differs from GRPO, etc is that the IS ratio is clipped directly rather than the fully policy clipping that is done in PPO derivatives. This introduces two new hyperparameters cispo_clip_ratio_high and cispo_clip_ratio_low to handle this clipping. They are each defaulted to 0.2.

Also, we've introduced a new policy loss function for CISPO, which is employed when adjusting the loss_mode in the run configuration. Altogether this looks like:

actor_rollout_ref.actor.policy_loss.loss_mode=cispo \
actor_rollout_ref.actor.policy_loss.cispo_clip_ratio_high=0.2 \
actor_rollout_ref.actor.policy_loss.cispo_clip_ratio_low=0.2 \

[nightlessbaron]

do we want to remove this?

[nightlessbaron]

please also add the following comment # Added by Reasoning360

[nightlessbaron]

imo, it would be ideal to create another directory called cispo instead of adding modifications in dapo