Scoring service

[metlos]

This PR stacks on top of #244 and introduces a service that performs the actual conversations with the judge LLMs.

Part of the PR stack:

• implement the database layer for session scoring #243
• introduce judge prompts and simple computed field to the SessionScore model #244
• Scoring service #245
• REST API controller for the scoring service #246
• Scoring UI #248

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

• 🔍 Trigger a full review

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch metlos/scoring-service

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[alexeykazakov]

Alert data can be anything. Not necessarily JSON. It's just a text.

[alexeykazakov]

I see this service duplicates a lot of functionality from the history service.
History service is a legacy name and we should probably rename it but it's essentially our main DB service.
So it works like this:
<Regular Service (like AlertService or ChatService)> -> History Service -> Repo(s) -> DB

The history service encapsulates high level DB logic like initialization, re-try, etc. All DB operations funnel through the history service. Let's keep it this way and move all these DB functionality to the history service.

I've created a PR to split the HistoryService into smaller sub-services: #250
So you can add a new subservice (and update the history service facade) there,

[alexeykazakov]

We need to move most of this functionality to the agent package. See the current structure of agents (base agent and extensions) and also agent controllers.

I would create a scoring agent, very similar to SynthesisAgent. And two controllers for it (very similar to Synthesis controllers). The rest of the functionality will be delegated to the corresponding llm clients.

We also would need a new configuration for chains. Similar to Synthesis configuration. Where we can define things like: llm provider, strategy (react, native-thining), etc.

Basically the scoring service becomes an orchestrator. It delegates DB work to the history service. Agent/LLM work to the Agent/Controllers. Session history building to the History Service. Prompt building to the prompt builder, etc.

[alexeykazakov]

I created a new method in the history service to generate the investigation session history: #249
You can use it as-is now. This is also used by the chat service so we don't have to duplicate it for different services/agents.