feat: Example for Integrating Realtime API into Sotopia. #234
Conversation
β¦a into feature/migrate-to-uv
Codecov Reportβ Patch coverage is @@ Coverage Diff @@
## main #234 +/- ##
==========================================
+ Coverage 74.02% 74.92% +0.89%
==========================================
Files 72 72
Lines 4628 4821 +193
==========================================
+ Hits 3426 3612 +186
- Misses 1202 1209 +7
... and 1 file with indirect coverage changes π New features to boost your workflow:
|
β¦ into feature/realtime-api
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
This PR is being reviewed by Cursor Bugbot
Details
You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.
To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.
| }, | ||
| ) | ||
| await self.websocket.__aenter__() | ||
| await self.websocket.send( |
There was a problem hiding this comment.
Bug: WebSocket connection entered twice
self.websocket = await connect(...) already yields an active ClientConnection in websockets>=13. Calling await self.websocket.__aenter__() again is inconsistent with the API and can raise at runtime or leave the connection in an unexpected state, breaking the realtime node initialization.
| for stream in stream_samples: | ||
| mixed_samples += stream | ||
| mixed_samples //= len(stream_samples) | ||
|
|
There was a problem hiding this comment.
Bug: Audio merge can divide by zero
merge_audio_streams divides by len(stream_samples) without guarding against streams being empty. A misconfigured AudioMixerNode with input_channels = [] (or any call site passing an empty list) will crash with a zero-division error during mixing.
| run: | | ||
| # Run this mypy instance against our main package. | ||
| uv run mypy --strict . | ||
| uv run --all-extras mypy --strict . |
There was a problem hiding this comment.
Bug: Mypy workflow no longer installs dependencies
The workflow removed uv sync and now runs uv run --all-extras mypy. If uv run doesnβt install [tool.uv].dev-dependencies (including mypy) in this CI context, the job can fail due to missing tools, and --all-extras can also force-install pyaudio, which is fragile in CI even with portaudio19-dev.
| yield ( | ||
| self.output_channel, | ||
| Message(data=Audio(audio=b"")), | ||
| ) # Unreachable code |
There was a problem hiding this comment.
Bug: Unreachable yield statements after raise
There are yield statements placed after raise ValueError in the event_handler methods of both AudioMixerNode and OpenAIRealtimeNode. These lines are unreachable code as noted by the comment "Unreachable code". This appears to be accidentally committed placeholder or debugging code that serves no purpose.
Additional Locations (1)
| assert isinstance( | ||
| action, AgentAction | ||
| ), f"Action must be AgentAction, got {type(action)}" | ||
| complied_actions[sender] = action |
There was a problem hiding this comment.
Bug: Type signature allows dict but implementation rejects it
The step and astep methods declare a type signature accepting dict[str, AgentAction] | dict[str, dict[str, int | str]], but the implementation asserts that actions must be AgentAction instances. This makes the second union type (dict[str, dict[str, int | str]]) impossible to use, contradicting the type annotation. The comment "Only validate dict inputs (from action space sampling)" also contradicts the assertion that rejects dict inputs.
Additional Locations (1)
| complied_actions[sender] = AgentAction.model_validate( | ||
| action_data, context=context | ||
| ) | ||
| context["sender"] = sender |
There was a problem hiding this comment.
Bug: Context variable built but never used
In both step and astep methods, a context dictionary is constructed with agent_names, available_action_types, and sender, but it's never passed to any validation or used elsewhere. This appears to be leftover code from an incomplete refactoring where the previous AgentAction.model_validate(action_data, context=context) call was removed but the context construction was not.
Additional Locations (1)
| if sys.version_info < (3, 11): | ||
| pass | ||
| else: | ||
| pass |
There was a problem hiding this comment.
Bug: Dead version check code with empty branches
The version check if sys.version_info < (3, 11): has empty pass statements in both branches, making it completely useless. This appears to be leftover code copied from realtime_websocket.py where similar version checks contain actual imports (from typing_extensions import Self vs from typing import Self), but the imports were removed or never added here.
There was a problem hiding this comment.
Bug: Subclass parse method incompatible with new context parameter
The PydanticOutputParser.parse method now accepts an optional context parameter, and agenerate passes context via **parse_kwargs when isinstance(output_parser, PydanticOutputParser) is True. However, EnvResponsePydanticOutputParser.parse overrides the parent's method with only a text parameter and no context. Since EnvResponsePydanticOutputParser inherits from PydanticOutputParser, the isinstance check passes, but calling parse(result, context=...) on it raises TypeError: got an unexpected keyword argument 'context'.
sotopia/generation_utils/output_parsers.py#L66-L74
sotopia/sotopia/generation_utils/output_parsers.py
Lines 66 to 74 in 360c11b
sotopia/generation_utils/generate.py#L593-L599
sotopia/sotopia/generation_utils/generate.py
Lines 593 to 599 in 360c11b
|
Looks like there are a few issues preventing this PR from being merged!
If you'd like me to help, just leave a comment, like Feel free to include any additional details that might help me get this PR into a better state. You can manage your notification settings |
Closes #229
π Description
This PR gives an example of simulating realtime conversation between agents in sotopia and aact.
β Checks
type/descript(e.g.feature/add-llm-agents)βΉ Additional Information
Note
Adds experimental realtime conversation examples and updates core Sotopia components with supporting server changes, tests, and mypy CI.
examples/experimental/realtime/{audio_mixer.py,input_node.py,realtime_websocket.py}.realtime_chat.toml,realtime_chat_human_in_the_loop.toml,speaker_listener.tomland areadme.md.sotopia/agents/llm_agent.py,sotopia/messages/message_classes.py.sotopia/envs/{evaluators.py,parallel.py}.sotopia/generation_utils/{generate.py,output_parsers.py}.sotopia/server.py..github/workflows/mypy.yml.pyproject.toml.tests/api/test_fastapi.py,tests/envs/{test_evaluators.py,test_parallel.py},tests/generation_utils/test_generation.py, with shared setup intests/conftest.py.Written by Cursor Bugbot for commit ab61a2b. This will update automatically on new commits. Configure here.