[FEAT]: Code appears ahead of documentation

[Babylonehy]

Description

Hi there,

thanks for the great work on Sotopia! I’m really excited to try it out.

I followed the instructions in the docs (via docs.sotopia.world) to run the evaluator, but I ran into a couple of issues:
• The example code in the documentation for the evaluator does not run successfully with the current codebase — some function or API appears to have changed.
• The gin configuration files referenced in the docs (or example folder) don’t seem to match what’s in the codebase any more: parameter names, config keys, and expected values seem misaligned.
• In short: it seems like the code has moved ahead of the documentation, so a user trying to follow the docs doesn’t get a working end-to-end setup.

Would you be able to clarify whether:
1. There is a plan to update the documentation (including examples and gin config) to match the current version of the code?
2. There is a “working example” (tested) branch or config that you recommend users use to get up and running?

Thanks again for your work — I look forward to getting this running and exploring it further!

Additional Information

No response

[XuhuiZhou]

Hi @Babylonehy, thanks for your points! We are actively maintaining Sotopia; however, we are running out of hands as all of the authors have different duties!

In short, we were hoping to update the documentation yet we could not promise an exact date.

To get started, have you tried the easy sample server in the readme?

[Babylonehy]

Thanks for the reply! Totally understand that keeping the docs up to date is not always easy.
Just a small suggestion — it would be really helpful if the documentation could indicate which code version or release it corresponds to. Even a brief note like “Docs for v0.x.x” would make things much more user-friendly.

[Babylonehy]

Description

Hi：

I’ve made some adjustments based on sotopia and was able to successfully run the benchmark. However, my current results are somewhat different from those reported in the paper (Sotopia-RL).

Here are my reproduced results (using GPT-4o as both agent and evaluator):

Setting	Believability	Relationship	Knowledge	Secret	Social Rules	Financial & Material Benefits	Goal	Overall Score	Setting Num	Episode Count
Sotopia-all	8.99 ± 0.01	2.53 ± 0.06	5.31 ± 0.13	-0.10 ± 0.04	-0.12 ± 0.03	0.60 ± 0.05	7.11 ± 0.10	3.47 ± 0.04	90.00 ± 0.00	562.00 ± 0.00
Sotopia-hard	8.88 ± 0.13	1.24 ± 0.22	4.55 ± 0.60	0.00 ± 0.00	−0.12 ± 0.16	0.42 ± 0.14	5.04 ± 0.31	2.86 ± 0.14	14	85

And here are the results from Sotopia-RL (using GPT-4o as partner):

Model	Setting	Goal	Average Score
GPT-4o	Sotopia-all	8.19	3.76
GPT-4o	Sotopia-hard	6.97	3.46

For the benchmark, I used the following command:

sotopia benchmark --only-show-performance \
  --models openrouter/openai/gpt-4o \
  --partner-model openrouter/openai/gpt-4o \
  --evaluator-model openrouter/openai/gpt-4o \
  --batch-size 10 \
  --task all \
  --push-to-db \
  --output-to-jsonl \
  --save-dir ./logs \
  --print-logs

The tag I’m using is v0.1.4.

Would you mind confirming:

1. Whether there were any specific hyperparameter or configuration differences (e.g., temperature, sampling strategy, or evaluator settings) between your released benchmark and the Sotopia-RL paper version?
2. Is there a recommended config file or branch for reproducing the Sotopia-RL results reported in Table 1?

Thanks again for your excellent work and for maintaining this project!

I also switch to v0.1.0-rc5，but got:

[XuhuiZhou]

Thanks for the reply! Totally understand that keeping the docs up to date is not always easy. Just a small suggestion — it would be really helpful if the documentation could indicate which code version or release it corresponds to. Even a brief note like “Docs for v0.x.x” would make things much more user-friendly.

That's a great point!

For reproducing the results, would you mind submitting a pull request so that we could tell what u changed exactly?

Also @lwaekfjlk @ProKil for the sotopia-RL related questions

[lwaekfjlk]

Hi @Babylonehy , based on my experience, it is possible that some episodes generated via sotopia are empty or incomplete due to some API rate limit. Could you check whether such incomplete or empty episode exist in your generated results?

[Babylonehy]

Hi @Babylonehy , based on my experience, it is possible that some episodes generated via sotopia are empty or incomplete due to some API rate limit. Could you check whether such incomplete or empty episode exist in your generated results?

Hi @lwaekfjlk,

Yes — after carefully reviewing the database and re-implementing the benchmark metric computation, I noticed that the main issue comes from duplicate scenes. The current benchmark code doesn’t seem to check for repeated simulations, and the fix_missing process also doesn’t account for cases where the same tag has been run multiple times.

As a result, some events in the storage_comb table appear missing or duplicated, and the total count exceeds 450 in certain cases. After filtering out these duplicated entries and re-running the benchmark, the results are now much closer to the reported ones — though still not exactly identical.

ps: ± is var not 95% ci

I’m not entirely sure whether the remaining gap between my reproduced results and the Sotopia-RL paper results can be considered acceptable.
Could it be mainly caused by differences in the temperature or other sampling parameters?

One quick question: since I’m using the same model (GPT-4o) for both agents and the evaluator, can I assume that it’s sufficient to run the benchmark once and simply take the average between the two agent roles?

but for deepseek-v3, I cannot reproduce the reuslt like Sotopia-RL. Lower than Sotopia-RL. I am not sure what happen to this. I only test on hard envs. And I also check the episode had right rewards output. I thought 14*5*2=140 is the expected episode numbers.

Thanks again for your clarification and support!

[lwaekfjlk]

Hi @Babylonehy, sorry for the late reply, and thanks for the detailed information!

Regarding the GPT-4o results: for the hard scenario, the goal score you reported is very close to ours, which suggests that the reproduction for that setting is accurate. The discrepancy in the “All” setting is likely due to how we compute the average. As noted in the evaluation settings section of Sotopia-RL, we run only two agent combinations per scenario across the full set of 90 social scenarios. This means our “All” score averages over 180 episodes, whereas yours averages over 450, which could naturally lead to numerical differences.

About hyperparameters: temperature and other sampling parameters have a substantial impact on performance. Could you share the hyperparameters you used for generation?

For the quick question, Yes, we do that as well.

For DeepSeek-V3, the results in Table 1 are taken directly from the paper (https://arxiv.org/html/2505.02156v1
). We did not run DeepSeek-V3 ourselves. Also, for the ablation experiments in Sotopia-RL, as stated on page 23 of Sotopia-RL, we use the DeepSeek-V3 model, not the chat version. This difference may matter, though we are not entirely sure how large the effect is.

Let me know if you need anything else! Happy to share more.

[lwaekfjlk]

For the sotopia-all results, it might worth checking whether all the conversations are still complete. It can have some sudden breakdown during generation and because the conversation is no longer complete, it can make the score lower than usual.

[Babylonehy]

Hi @Babylonehy, sorry for the late reply, and thanks for the detailed information!

Regarding the GPT-4o results: for the hard scenario, the goal score you reported is very close to ours, which suggests that the reproduction for that setting is accurate. The discrepancy in the “All” setting is likely due to how we compute the average. As noted in the evaluation settings section of Sotopia-RL, we run only two agent combinations per scenario across the full set of 90 social scenarios. This means our “All” score averages over 180 episodes, whereas yours averages over 450, which could naturally lead to numerical differences.

About hyperparameters: temperature and other sampling parameters have a substantial impact on performance. Could you share the hyperparameters you used for generation?

For the quick question, Yes, we do that as well.

For DeepSeek-V3, the results in Table 1 are taken directly from the paper (https://arxiv.org/html/2505.02156v1 ). We did not run DeepSeek-V3 ourselves. Also, for the ablation experiments in Sotopia-RL, as stated on page 23 of Sotopia-RL, we use the DeepSeek-V3 model, not the chat version. This difference may matter, though we are not entirely sure how large the effect is.

Let me know if you need anything else! Happy to share more.

Hi @lwaekfjlk,

Thanks for the clarification — that definitely helps explain the discrepancy in the “All” setting. Averaging over 180 vs. 450 episodes would indeed lead to numerical differences.

Regarding hyperparameters, I ran a set of controlled experiments to assess the impact of temperature, and the results show notable performance differences.

---

🔧 Experimental Setup

I conducted 5 independent runs per temperature setting to ensure statistical reliability.

The temperature parameter was changed directly in the agent generation function:

action = await agenerate_action(
    self.model_name,
    history="\n".join(f"{y.to_natural_language()}" for x, y in self.inbox),
    turn_number=obs.turn_number,
    action_types=obs.available_actions,
    agent=self.agent_name,
    goal=self.goal,
    temperature=0.0,
    script_like=self.script_like,
)

All experiments used the standard evaluation command:

python examples/experiment_eval.py \
  --gin_file sotopia_conf/generation_utils_conf/generate.gin \
  --gin_file sotopia_conf/server_conf/server.gin \
  --gin_file sotopia_conf/run_async_server_in_batch.gin \
  "--gin.ENV_IDS=${ENV_IDS}" \
  "--gin.ENV_MODEL='${ENV_MODEL}'" \
  "--gin.AGENT1_MODEL='${AGENT1_MODEL_NAME}'" \
  "--gin.AGENT2_MODEL='${AGENT2_MODEL_NAME}'" \
  --gin.BATCH_SIZE=20 \
  "--gin.TAG='${RUN_TAG}'" \
  "--gin.TAG_TO_CHECK_EXISTING_EPISODES='${RUN_TAG}'" \
  --gin.PUSH_TO_DB=True \
  --gin.VERBOSE=False \
  --gin.OMNISCIENT=False \
  --gin.LITE=False

---

📊 Temperature Impact Analysis

All Episodes (Average of 5 runs)

Metric	Temp = 0.0	Default Temp	Δ (Difference)
believability	8.98	8.92	+0.06 ✓
financial_and_material_benefits	0.60	0.56	+0.04
knowledge	4.74	4.66	+0.08
relationship	2.46	2.23	+0.23 ✓
goal	7.45	7.00	+0.45 ✓
overall_score	3.45	3.32	+0.13 (+3.9%)

❓ Questions

1. Which temperature setting was used in the official Sotopia / Sotopia-RL benchmarks?
   The difference in overall_score and goal suggests temperature may significantly affect reproducibility.

2. What are the recommended hyperparameters for reproduction?
   Specifically:

   • Temperature (0 vs. default / 1.0?)
   • Top-p
   • Top-k
   • Any other sampling parameters used in official evaluations

3. Should temperature = 0 be considered the standard for benchmark evaluation, given its stability and higher average performance?

Thx for your patience!