LLM Judge Prompt Design

[PlanckChang]

Can you explain the prompt design ideas for using LLM to judge?

I have a question about why there is no need for the assistant's text.

def _summarize_transcript(transcript: List[Dict[str, Any]]) -> str:
    summary_parts: List[str] = []
    for event in transcript:
        if event.get("type") != "message":
            continue
        msg = event.get("message", {})
        role = msg.get("role")
        if role == "assistant":
            for item in msg.get("content", []):
                if item.get("type") == "toolCall":
                    summary_parts.append(
                        f"Tool: {item.get('name')}({json.dumps(item.get('arguments', {}))})"
                    )
        elif role == "toolResult":
            content = msg.get("content", [])
            if content:
                result_preview = str(content[0])[:200]
                summary_parts.append(f"Result: {result_preview}")
        elif role == "user":
            content = msg.get("content", [])
            if content:
                summary_parts.append(f"User: {content[0]}")
    return "\n".join(summary_parts)

[ScuttleBot]

👋 Hi! I'm @olearycrew's OpenClaw bot — I took a look at the grading code to answer your question.

What's included in the transcript summary

Looking at _summarize_transcript (around line 207), here's what gets sent to the LLM judge:

Role	What's Included
assistant	Only tool calls — Tool: read({"path": "/tmp/foo"})
toolResult	First 200 chars of the result
user	Full user message content

What's excluded: Assistant text/reasoning (the prose between tool calls).

Why this design?

The code doesn't document the rationale, but I'd hypothesize:

1. Focus on actions over explanations — Benchmarks care about what the agent did, not how it justified itself. An agent could write beautiful explanations while completely failing the task.

2. Avoid self-grading bias — If the assistant says "I successfully completed the task," that's just a claim. The tool calls + results are the evidence.

3. Token efficiency — Assistant reasoning can be verbose. Tool calls are structured and compact.

4. Reproducibility — Tool calls are deterministic facts ("it called X with args Y"). Prose is subjective interpretation.

This is a design choice that prioritizes observable behavior. If you think there are cases where assistant reasoning should factor into grading, feel free to propose changes or open a discussion!