How can I reproduce `snowball` and `recap`?

[GwonHyeokMIN]

I would like to reproduce the values from Table 2 of the paper, but I’m not sure how to do so. I would appreciate any guidance you could provide.

[rooa]

Hey @GwonHyeokMIN , thanks for the interest! For each, we have simulator_recap.py and simulator_snowball.py that does the job. After setting the env, you should be able to run gpt-4o-mini simulation on the Code task with

• snowball:

python simulator_snowball.py --task code --assistant_model gpt-4o-mini

• recap: We leverage the sharded simulations and then add one more "recap" turn. So it will be

# sharded run
python run_simulations.py \
  --N_sharded_runs 1 \
  --N_full_runs 0 \
  --N_concat_runs 0 \
  --tasks code \
  --assistant_model gpt-4o-mini

# load logs from the sharded runs and simulate one more turn
python simulator_recap.py --task code --models gpt-4o-mini

[GwonHyeokMIN]

The following errors occur when I run Snowball and Recap, respectively.

• snowball:

python simulator_snowball.py --task code --assistant_model gpt-4o-mini
usage: simulator_snowball.py [-h] [--task {database,wmt,summary,math,actions,data2text,python}] [--assistant_model ASSISTANT_MODEL] [--system_model SYSTEM_MODEL] [--user_model USER_MODEL] [--N_snowball_runs N_SNOWBALL_RUNS]
                             [--N_workers N_WORKERS] [--verbose] [--models MODELS [MODELS ...]]
simulator_snowball.py: error: argument --task: invalid choice: 'code' (choose from 'database', 'wmt', 'summary', 'math', 'actions', 'data2text', 'python')

• recap

python simulator_recap.py --task code --models gpt-4o-mini
Traceback (most recent call last):
  File "/home/ghm1212/5_29/lost_in_conversation/simulator_recap.py", line 83, in <module>
    samples = get_task(task_name).get_samples(filter="full")
TypeError: get_samples() got an unexpected keyword argument 'filter'

[rooa]

Thanks for the report. Just pushed the fixes for those.