New idea: Multi-turn & Compositional Reasoning Evaluation

[sharmaanchita]

Problem
The system only supports single-shot prompt-response evaluation, missing critical real-world reasoning capabilities.

Basis of issue

1. Multi-turn conversation evaluation
2. Compositional task design (chained reasoning)
3. Logical consistency / trace quality metrics
4. Stateful prompt handling

Importance

1. Real-world AI usage is multi-turn
2. Compositional reasoning is a core capability
3. Modern benchmarks already evaluate this

Current implementation gap

1. Single prompt → single response only

Implementation checklist

1. Support for multi-step prompt sequences
2. Scoring based on reasoning consistency across turns
3. Optional evaluation of intermediate reasoning quality
4. Backward compatibility with single-shot prompts

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Related Issues

• Enhancement: Automatic Rolling Reservoir & Prompt Retirement #58
• Enhancement: Synchronized Cohort-Based Evaluation #60
• Implement Prompt Versioning with Immutable Benchmark References and Result Diffing #53
• New idea: Sealed Execution Environment for Model Evaluation #59
• Benchmark leaderboards become hard to interpret when most models score near-perfect #48

👤 Suggested Assignees

• sharmaanchita
• SanthoshPradeep
• Meghhanaa

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!

[reisepass]

yea for sure we already have some code for multi-turn just have not made it public yet. We are working on improving the data schema to better accommodate all types of test cases including multi turn chat .

I could show you some of the unpublished code for evaluating multi-turn an issue that has come up is that the complexity of the LLM Judge goes up so the LLM Judge score gets a higher variance and can just make bad judgments more often than comparing to 1 single expected response.

Have you done multi turn LLM evaluations before ? If so any ideas on how to get the judge to give more consistent scores ?

[sharmaanchita]

I’ve read through a lot of the recent work on LLM-as-judge and multi-turn evaluation, and the variance spike you’re seeing seems pretty expected once teams move past single expected answers.

Without seeing the code, my guess is the remaining variance is structural - the judge is effectively enforcing an implicit “ideal” conversational path and penalizing reasonable alternatives. Recent research seems to point toward shifting away from single-trajectory matching and more toward constraint, consistency, or relative-based evaluation, rather than trying to fully stabilize one holistic judge score.