Dynamic Contextual RL

A framework for reinforcement learning in environments where context (physics parameters, goals, etc.) can change dynamically during episodes. This extends the contextual RL paradigm by allowing context to evolve over time according to configurable update functions (e.g. sinusoidal, piecewise constant, random walk).

Note: Experiments will be populated here as they become ready. For now, this README focuses on getting the examples running.

Installation

Create a virtual environment and install the core dependencies:

pip install -r requirements/requirements_dcrl.txt

To install CARL, you need to run a separate command:

git clone https://github.com/automl/CARL.git --recursive
cd CARL
pip install .

The CartPole, Walker, and Vehicle Racing examples use Gymnasium and Box2D.

The Go2 and Quadruped examples require additional dependencies: dm_control, mujoco (already included in requirements_dcrl.txt)

Running the Examples

Examples live under dynamic_crl/examples/. Run them from the project root so imports resolve correctly.

Dynamic CartPole

Context (e.g. pole length) changes over time while the agent interacts:

python dynamic_crl/examples/dynamic_cartpole/visualize_dyn_cp.py

Dynamic Walker

Bipedal walker with a dynamically moving payload (COM_X):

python dynamic_crl/examples/dynamic_walker/visualize_dyn_walker.py

Dynamic Vehicle Racing

Racing car with a payload that slides left/right over time:

python dynamic_crl/examples/dynamic_vehicle_race/visualize_dyn_racer.py

Drive the car yourself while the payload moves!

python dynamic_crl/examples/dynamic_vehicle_race/human_control_dyn_racer.py

Go2 & Quadruped

These use MuJoCo/dm_control, the Go2 environment comes with all the assets in this repo:

python dynamic_crl/examples/dynamic_go2/dynamic_go2_vis.py
python dynamic_crl/examples/dynamic_quadruped/visualize_ctx_quad.py

Project Structure

dynamic_crl/
├── examples/          # Runnable demos per environment
└── src/
    ├── dynamic_carl/  # Core wrappers & context update logic
    ├── gym_envs/      # Gymnasium envs (cartpole, walker, vehicle)
    ├── dmc_envs/      # dm_control/MuJoCo envs (go2, quadruped)
    └── utils/         # Factories, accessors, helpers
└── helix_utils/       # Cluster submission utilities

experiments/           # Experiments code per environment
├── car_racing/
├── cartpole/
└── walker/

Context Update Functions

The framework provides several built-in update modes in gym_context_updates:

• make_sinusoidal — oscillating context
• make_random_walk — stochastic drift
• make_piecewise_constant — step changes at intervals
• make_continuous_incrementer — linear ramp
• make_cosine_annealing — smooth annealing
• make_sudden_jump — discrete jumps

Each example configures these for the relevant context features (e.g. pole length, payload position).