Runtime 500: ConversationInfo validation error (missing id/agent/workspace)

[simonrosenberg]

Summary

During SWT-bench eval (attempt 2), the eval runtime returned a 500 on GET /api/conversations/{id}. The server log shows a pydantic_core.ValidationError when constructing ConversationInfo, with required fields id, agent, and workspace missing from the input dict. The run continued afterward and eventually completed, but this 500 is a runtime instability and should be addressed.

Context

• Benchmark: swtbench
• Eval run: eval-20613780379-claude-son
• Conversation id: 9f538871-e9c7-4bd1-908c-25fc1050867e
• Runtime id: zaerpphmsmedfvor
• Pod: runtime-zaerpphmsmedfvor-78cf77bb79-8b8sj
• Time (UTC): 2025-12-31 06:56:57

Observed behavior

GET /api/conversations/9f538871-e9c7-4bd1-908c-25fc1050867e returned HTTP 500. The error is a ConversationInfo validation failure due to missing id, agent, and workspace.

Log excerpt (gcloud logs)

Unhandled exception on GET /api/conversations/9f538871-e9c7-4bd1-908c-25fc1050867e
pydantic_core._pydantic_core.ValidationError: 3 validation errors for ConversationInfo
id
  Field required [type=missing, input_value={'title': None, 'metrics'...=datetime.timezone.utc)}, input_type=dict]
agent
  Field required [type=missing, input_value={'title': None, 'metrics'...=datetime.timezone.utc)}, input_type=dict]
workspace
  Field required [type=missing, input_value={'title': None, 'metrics'...=datetime.timezone.utc)}, input_type=dict]

Likely source

• openhands-agent-server/openhands/agent_server/conversation_service.py _compose_conversation_info()
  • ConversationInfo(**state.model_dump(), title=..., metrics=..., created_at=..., updated_at=...)
  • The state.model_dump() appears to be missing id/agent/workspace at the moment of the request.

Additional details

• A few seconds before the 500, the runtime logged “Created new conversation ...” with a full state that included id/agent/workspace.
• After this single 500, subsequent GETs returned 200 and the run completed. This suggests a transient or race condition where state.model_dump() can be incomplete.

Suggested direction

• Investigate why ConversationState.model_dump() can omit required fields (id, agent, workspace).
• Consider hardening _compose_conversation_info() to fall back to stored.id/agent/workspace or return a 503 with retryable semantics instead of 500.

References

• Runtime 500 in swtbench attempt 2 (eval-20613780379-claude-son)
• Conversation archive: gs://openhands-evaluation-results/artifacts/eval-20613780379-claude-son/conversations/sympy__sympy-23824.tar.gz

[simonrosenberg]

Findings (based on logs + code):

• The 500 is a transient state snapshot race, not real removal of fields. In the runtime logs the POST /conversations/{id}/run happens at 06:56:57.249Z and GET /conversations/{id} fails at 06:56:57.448Z, then 200s immediately after (see runtime_zaerpphmsmedfvor_065650_065705.json). This is the tight window right after run starts.
• GET path calls state.model_dump() without locking in openhands/agent_server/conversation_service.py and run starts a background thread that mutates execution_status under the state lock. That mutation triggers autosave (ConversationState.__setattr__ -> model_dump_json) while GET is serializing, so Pydantic can emit a partial dict (missing id/agent/workspace).

On the PR fix: locking ConversationState.model_dump() does prevent torn snapshots, but it also means GET blocks on the same lock used for the entire agent.step() call. LocalConversation.run() holds the state lock while calling agent.step() (LLM + tools), so GET can now block for seconds.

Issues with current solution and what we should change:

1. Mutations are not enforced to be locked. setattr doesn’t acquire the lock; it assumes callers use with state:. One lock-free write reintroduces the race. We should either lock inside setattr (reentrant FIFO lock makes this safe) or assert state._lock.owned() when autosave is enabled so any lock-free mutation fails loudly.
2. In-place updates bypass setattr entirely. Examples: ConversationStats.register_llm() mutates usage_to_metrics in place; SecretRegistry.update_secrets() mutates in place. These need to go through state-level mutator methods that acquire the lock.
3. Don’t rely on OpenHandsModel.model_dump_json() calling self.model_dump() for lock coverage. Override ConversationState.model_dump_json() to lock explicitly.
4. Reduce lock hold time. The state lock is held across agent.step() (LLM + tool execution). That’s too coarse. We should narrow the lock scope: take a snapshot of the fields agent.step() needs under lock, run LLM/tools without the lock, then reacquire to apply updates. This avoids GET blocking for long steps.

Plan proposal:

• Add ConversationState.snapshot() (locked) to copy the fields agent.step() needs (events, confirmation_policy, analyzer, execution_status, etc.).
• Refactor LocalConversation.run() to: lock -> snapshot -> unlock -> LLM/tool work -> lock -> apply state updates / emit events.
• Add state-level mutators (update_secrets, update_stats, etc.) that lock internally and update call sites.
• Enforce mutation locking in setattr (lock or assert).
• Override ConversationState.model_dump_json() to lock.
• Add a concurrency test that runs a mutation + GET snapshot in parallel and asserts required fields are present; also add a test that GET does not block on long tool execution (can be simulated with a sleep tool).

[enyst]

Sorry I’m dense, why the fields id, agent, and workspace? I don’t figure it out from that error?

On another note, workspace is strange, how can that be required, don’t we aim to pass from local to remote

[simonrosenberg]

Sorry I’m dense, why the fields id, agent, and workspace? I don’t figure it out from that error?

On another note, workspace is strange, how can that be required, don’t we aim to pass from local to remote

gcloud logs

Unhandled exception on GET /api/conversations/9f538871-e9c7-4bd1-908c-25fc1050867e
pydantic_core._pydantic_core.ValidationError: 3 validation errors for ConversationInfo
id
  Field required [type=missing, input_value={'title': None, 'metrics'...=datetime.timezone.utc)}, input_type=dict]
agent
  Field required [type=missing, input_value={'title': None, 'metrics'...=datetime.timezone.utc)}, input_type=dict]
workspace
  Field required [type=missing, input_value={'title': None, 'metrics'...=datetime.timezone.utc)}, input_type=dict]

[simonrosenberg]

@OpenHands
Read this issue and read the associated PR diff #1558
The issue is actually quite complicated.
Can we start with a simple workaround?
Before creating ConversationInfo, check that all required fields are there. If not, wait for 0.1s and redump the conversation state. If not, wait 0.3s and redump. If not: raise an error
That should do the trick, right? Add a comment with a TODO and a link to this issue so we can do a better fix next time
#1558

[openhands-ai]

I'm on it! simonrosenberg can track my progress at all-hands.dev

[simonrosenberg]

@enyst doing a cheap fix because a real fix is above my current bandwidth :p

[openhands-ai]

Summary

I've implemented the requested workaround for issue #1557 (Runtime 500: ConversationInfo validation error).

Changes Made

1. openhands-agent-server/openhands/agent_server/conversation_service.py:

• Added _get_state_dump_with_retry() function that:
  • Checks if all required fields (id, agent, workspace) are present in the state dump
  • If missing, waits 0.1s and retries
  • If still missing, waits 0.3s and retries
  • If still missing after all retries, raises a RuntimeError with a helpful message
• Added a TODO comment with a link to issue Runtime 500: ConversationInfo validation error (missing id/agent/workspace) #1557 for future proper fix
• Modified _compose_conversation_info() to use the new retry function

2. tests/agent_server/test_conversation_service.py:

• Added 4 unit tests for _get_state_dump_with_retry():
  • Returns immediately when all fields are present
  • Retries once (0.1s) and succeeds
  • Retries twice (0.1s, 0.3s) and succeeds
  • Raises RuntimeError after all retries exhausted

Checklist

• ✅ Check required fields before creating ConversationInfo
• ✅ Wait 0.1s and retry if fields missing
• ✅ Wait 0.3s and retry if still missing
• ✅ Raise error if still missing after retries
• ✅ Added TODO comment with link to issue Runtime 500: ConversationInfo validation error (missing id/agent/workspace) #1557
• ✅ All pre-commit checks pass
• ✅ All 45 tests pass

PR Created

Draft PR: #1559

The PR fixes #1557 with a simple workaround as requested, while leaving a TODO for the proper locking fix proposed in PR #1558.

View full conversation

[xingyaoww]

@OpenHands can you look into this issue and analyze the root cause of it? Do not modify any code yet. Why would the race condition happen

[openhands-ai]

I'm on it! xingyaoww can track my progress at all-hands.dev