🚀[Feature Request]: Adding CI Routing for command /review to provide feedback for PRs #96
Description
A clear and concise description of what you want to happen.
Right now most AI-assisted PR review tools rely on calling an LLM for every PR review. However, many checks could be handled deterministically inside CI using tools (tests, linting, reproducibility checks, documentation presence, etc.).
I was thinking it might be useful to introduce a lightweight rubric-based CI workflow combined with a simple routing or decision layer.
Instead of running all checks or sending the entire diff to an LLM, the system could first analyze the PR (diff size, changed file types, modified directories, etc.) and route it to the appropriate tools. For example:
Changes in training/ → run reproducibility and experiment config checks
Changes in docs/ → run documentation validation
Changes in src/ → run tests and linting
Large structural changes → optionally trigger an LLM-based review
This could be implemented with a simple configuration file (e.g., .rubric.yml) defining evaluation criteria and routing rules. CI would run the relevant tools and produce a structured rubric-style report on the PR.
An LLM could optionally be used afterward to summarize the results or provide higher-level feedback, but the core evaluation would remain deterministic and tool-driven.
This approach could make PR feedback faster, cheaper, and more structured, while still allowing deeper reasoning when needed. Curious if something like this would fit into the current CI or agent workflow design.
A clear and concise description of any alternative solutions or features you've considered, if any.
No response
Additional context
No response