Parse new LCORE tool_call format #150
Open
+118
−13
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
WalkthroughThe streaming parser has been extended to handle tool_result events alongside tool calls. A new tool_calls_by_id mapping tracks tool calls by ID, enabling result association. Parsing now supports both legacy (tool_name/arguments) and new (name/args) formats. A new helper processes and applies results to corresponding tool calls. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Author
|
@asamal4 w/o this change i was not getting tool_call data in the eval results when using a current LCORE deployment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@src/lightspeed_evaluation/core/api/streaming_parser.py`:
- Around line 206-219: The parser incorrectly treats an empty new-format args
dict as missing because it uses "or" when assigning tool_name and arguments;
change the logic to prefer the "name"/"args" keys when present by checking key
presence on event_data (e.g., use "if 'name' in event_data: tool_name =
event_data['name'] else: tool_name = event_data.get('tool_name')" and similarly
for "args"/"arguments") and then treat arguments as missing only if neither key
is present (i.e., arguments is None and neither "args" nor "arguments" keys
exist). Update the code paths around event_data, tool_name, and arguments in
streaming_parser.py so empty dicts for args are accepted as valid.
Comment on lines
206
to
219
| try: | ||
| tool_name = token.get("tool_name") | ||
| arguments = token.get("arguments") | ||
| # Support both "name"/"args" (new format) and "tool_name"/"arguments" (legacy) | ||
| tool_name = event_data.get("name") or event_data.get("tool_name") | ||
| arguments = event_data.get("args") or event_data.get("arguments") | ||
|
|
||
| if not tool_name: | ||
| logger.debug("Tool call missing tool_name field") | ||
| logger.debug("Tool call missing name/tool_name field") | ||
| return None | ||
|
|
||
| # Only process tool calls that explicitly have arguments field | ||
| # Arguments can be empty dict, but field should exist | ||
| if arguments is None: | ||
| logger.debug("Tool call missing arguments field for %s", tool_name) | ||
| logger.debug("Tool call missing args/arguments field for %s", tool_name) | ||
| return None | ||
|
|
Contributor
There was a problem hiding this comment.
Handle empty new-format args without dropping tool calls.
Line 208-210 uses or, so args={} in the new format becomes falsy and is treated as missing, causing valid tool calls to be discarded. Consider key-presence checks instead.
🛠️ Proposed fix
- tool_name = event_data.get("name") or event_data.get("tool_name")
- arguments = event_data.get("args") or event_data.get("arguments")
+ tool_name = (
+ event_data.get("name") if "name" in event_data else event_data.get("tool_name")
+ )
+ if "args" in event_data:
+ arguments = event_data.get("args")
+ elif "arguments" in event_data:
+ arguments = event_data.get("arguments")
+ else:
+ arguments = None🤖 Prompt for AI Agents
In `@src/lightspeed_evaluation/core/api/streaming_parser.py` around lines 206 -
219, The parser incorrectly treats an empty new-format args dict as missing
because it uses "or" when assigning tool_name and arguments; change the logic to
prefer the "name"/"args" keys when present by checking key presence on
event_data (e.g., use "if 'name' in event_data: tool_name = event_data['name']
else: tool_name = event_data.get('tool_name')" and similarly for
"args"/"arguments") and then treat arguments as missing only if neither key is
present (i.e., arguments is None and neither "args" nor "arguments" keys exist).
Update the code paths around event_data, tool_name, and arguments in
streaming_parser.py so empty dicts for args are accepted as valid.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds support for the current LCORE tool_call response format, e.g:
Type of change
Tools used to create PR
Identify any AI code assistants used in this PR (for transparency and review context)
Related Tickets & Documents
Checklist before requesting a review
Testing
Verified by running evals against a current LCORE deployment (v0.4+)
Summary by CodeRabbit
Release Notes
New Features
Tests
✏️ Tip: You can customize this high-level summary in your review settings.