Skip to content

add multi queries and return union of different LLM returns#12

Open
rfievet wants to merge 3 commits intomainfrom
Robin--addMulti_query_union_results_to_improve_consistency
Open

add multi queries and return union of different LLM returns#12
rfievet wants to merge 3 commits intomainfrom
Robin--addMulti_query_union_results_to_improve_consistency

Conversation

@rfievet
Copy link
Copy Markdown
Contributor

@rfievet rfievet commented Jul 8, 2025

add multi queries and return union of different LLM returns (with a 5 max attempt).

there is still some minor divergence but less than before.
I didn't want to add more than 5 to not burn too much tokens

@rfievet rfievet requested a review from ketanbj July 8, 2025 19:49
Comment thread softwarecitationauditor/checker.py Outdated
final_json = json.loads(history[-1])
except Exception:
logger.warning("Final step result is not valid JSON")
return {"attempts": all_attempts, "result": final_json} No newline at end of file
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there something that deduplicates results in all_attempts or will that happen somewhere else?

Comment thread softwarecitationauditor/checker.py Outdated
templates = prompt_template.split("---SPLIT STEP---")

history = []
all_attempts = []
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this is a list of lists (type annotations would help), but it seems like all_attempts will only ever have 1 element. Am I reading the code correctly? If so, do we need both all_attempts and attempts? If we don't need both, would it be better to call the list results given that it is made up of result objects?

Copy link
Copy Markdown
Collaborator

@ketanbj ketanbj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants