feat: add Symphony integration with orchestration scoring#15
feat: add Symphony integration with orchestration scoring#15WellDunDun merged 10 commits intomasterfrom
Conversation
Add framework detection, workflow config scanning, and orchestration readiness fields to AuditRuntimeContext. Bump max_score to 22 to reflect the new agent_workflow orchestration point. Add two new detection helpers (checkWorkflowConfigForPattern, checkWorkflowsForMergeProtection) for scanning workflow configs and CI pipelines. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add workflow config, skills directory, and merge protection checks to the doctor command for validating Symphony-ready project structure. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scaffold WORKFLOW.md (agent orchestration config) and SPEC.md (behavioral specification) during reins init, with a next-step hint to customize workflow settings. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add scoreAgentWorkflowOrchestration() to award a point when 3+ orchestration signals are detected (workflow config, skills directory, isolation policy, concurrency limits, merge protection). Bumps agent_workflow max from 4 to 5 and recalibrates maturity thresholds across scoring and evolve to account for the expanded 22-point scale. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… thresholds Add workflow configuration (L1 step 6), composable skills directory (L2 step 7), and workspace isolation policy (L3 step 8) to evolution paths. Adjust resolveCurrentLevelKey boundaries to match new maturity calibration (L1<=11, L2<=16, L3<=19). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Align all documentation with new 0-22 scoring range, recalibrated maturity levels (L0-L4), and expanded agent_workflow dimension (0-5). Add symphony-integration.md design decision document. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
📝 WalkthroughWalkthroughExtends audit max score from 18→22; adds orchestration-readiness detection and frameworks detection; increases agent_workflow max to 5; renames maturity levels and updates thresholds; adds WORKFLOW.md and SPEC.md templates; introduces new doctor checks and evolve steps; bumps package versions to 0.1.4. Changes
Sequence DiagramsequenceDiagram
participant Scanner as File Scanner
participant Context as Context Builder
participant Detector as Framework Detector
participant Scorer as Scoring Engine
participant Output as Audit Result
Scanner->>Context: Read repo files (WORKFLOW.md, workflow.yml, .codex/, SPEC.md, skills dirs, policies, workflows)
Context->>Detector: Probe for framework markers (symphony, claude-code, conductor, codex)
Detector-->>Context: frameworksDetected[]
Context-->>Scorer: AuditRuntimeContext (flags + frameworksDetected)
Scorer->>Scorer: Compute orchestrationSignals from flags:
Scorer->>Scorer: hasWorkflowConfig, hasSkillsDirectory, hasIsolationPolicy, hasConcurrencyLimits, hasMergeProtection
alt orchestrationSignals >= 3
Scorer->>Scorer: Increment agent_workflow score
Scorer->>Output: Add orchestration readiness finding
else orchestrationSignals >= 1
Scorer->>Output: Add partial signals finding
end
Scorer->>Scorer: Compute totalScore and maturity_level (max_score=22)
Scorer-->>Output: Return AuditResult (scores, maturity_level, frameworks_detected)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a16b0f0c26
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Actionable comments posted: 7
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
package.json (1)
8-8:⚠️ Potential issue | 🟡 MinorPossible typo in
audit:selfscript path.The path
../...appears to have an extra dot. Based on learnings, the self-audit command should use../..to target the repository root.🔧 Proposed fix
- "audit:self": "cd cli/reins && bun src/index.ts audit ../...", + "audit:self": "cd cli/reins && bun src/index.ts audit ../..",🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@package.json` at line 8, The "audit:self" npm script uses an incorrect path token "../..." — update the script value for "audit:self" to use "../.." so the command cd cli/reins && bun src/index.ts audit ../.. correctly targets the repository root; locate the "audit:self" entry in package.json and replace the "../..." portion with "../..".README.md (1)
193-212:⚠️ Potential issue | 🟡 MinorAdd a language to this fenced block.
This trips markdownlint
MD040and can fail docs linting.textorplaintextis enough for the scaffold tree.Suggested fix
-``` +```text AGENTS.md # Concise map (~100 lines) for agents ARCHITECTURE.md # Domain map, layer rules, dependency direction WORKFLOW.md # Agent workflow and orchestration config SPEC.md # Project specification and constraints risk-policy.json # Risk tiers + docs drift rules (policy-as-code) docs/ golden-principles.md # Mechanical taste rules enforced in CI design-docs/ index.md # Design doc registry with verification status core-beliefs.md # Agent-first operating principles product-specs/ index.md # Product spec registry exec-plans/ active/ # Currently executing plans completed/ # Historical plans with outcomes tech-debt-tracker.md # Known debt with priority and ownership references/ # External LLM-friendly reference docs generated/ # Auto-generated docs (schema, API specs)</details> <details> <summary>🤖 Prompt for AI Agents</summary>Verify each finding against the current code and only fix it if needed.
In
@README.mdaround lines 193 - 212, The fenced code block that lists files
(starting with "AGENTS.md", "ARCHITECTURE.md", etc.) is missing a language tag
and triggers markdownlint MD040; change the opening fence fromto a tagged fence such astext (orplaintext) so the block is recognized as plain text and the linter stops complaining—update the README.md fenced block that contains AGENTS.md/ARCHITECTURE.md/WORKFLOW.md to usetext.</details> </blockquote></details> </blockquote></details>🧹 Nitpick comments (2)
cli/reins/src/lib/commands/init.ts (1)
115-122: Consider reordering the WORKFLOW.md review step.The step to review
WORKFLOW.mdis placed after "Run 'reins audit .' to see your starting score", but users would typically want to customize workflow settings before running the initial audit. Consider moving it earlier in the list.♻️ Suggested reordering
const steps = [ "Edit AGENTS.md — fill in the project description", "Edit ARCHITECTURE.md — define your business domains", "Review risk-policy.json — set tiers and docs drift rules for your repo", "Edit docs/golden-principles.md — customize rules for your project", + "Review WORKFLOW.md — customize agent behavior and orchestration settings", "Run 'reins audit .' to see your starting score", - "Review WORKFLOW.md — customize agent behavior and orchestration settings", ];🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@cli/reins/src/lib/commands/init.ts` around lines 115 - 122, The ORDER of steps in the const steps array in init.ts places "Review WORKFLOW.md — customize agent behavior and orchestration settings" after the "Run 'reins audit .'" step; move the WORKFLOW.md entry earlier (for example, immediately before "Run 'reins audit .'" or after "Review risk-policy.json") so users customize workflow settings before running the initial audit, updating the steps array accordingly.cli/reins/src/index.test.ts (1)
179-190: These assertions still don't cover the new scoring contract.The PR changes orchestration scoring,
frameworks_detected, and the 5/6, 11/12, 16/17, 19/20 cutovers, but this file only updatesmax_score. The"returns correct maturity levels at boundaries"case still doesn't exercise any boundary values. Please add at least one 2-signal vs 3-signal orchestration fixture and direct threshold assertions.Also applies to: 258-272
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@cli/reins/src/index.test.ts` around lines 179 - 190, The tests need to be updated to exercise the new orchestration scoring and explicit cutover thresholds: add at least one fixture directory (e.g., tmpDir("audit-orch-2sig") and tmpDir("audit-orch-3sig")) that produces 2 vs 3 orchestration signals, call runCli(`audit ${dir}`) and assert the orchestration-related fields (for example result.scores.orchestration.score and result.scores.orchestration.frameworks_detected) to show the 2→3 signal difference, and update the "returns correct maturity levels at boundaries" test to include direct assertions that specific total_score values hit the maturity cutoffs (5 vs 6, 11 vs 12, 16 vs 17, 19 vs 20) by constructing inputs or fixtures that produce totals at those thresholds and asserting maturity_level and total_score accordingly; touch the tests that reference runCli, tmpDir, result.total_score, result.maturity_level, and result.scores.* to locate where to add these checks.🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed. Inline comments: In `@cli/reins/src/lib/audit/context.ts`: - Around line 56-63: The audit result never copies detected frameworks into result.frameworks_detected; update runAudit() to assign the runtime context's frameworks list to the result after scoring is applied by adding result.frameworks_detected = context.frameworksDetected (ensure you reference the same context variable used when frameworks are collected and do this after applyAuditScoring() completes so the final JSON contains the detected frameworks). In `@cli/reins/src/lib/commands/evolve.ts`: - Around line 99-105: The evolve step object with action "Define workflow configuration" currently has automated: false but WORKFLOW.md is scaffolded by the reins init flow; change automated to true so evolve's capabilities match `reins init` (i.e., mark that step as automatable), and also update the init-backfill/required-artifacts logic to treat WORKFLOW.md and SPEC.md as required artifacts (wherever the init-backfill check enumerates required files) so repos that miss these files get them backfilled automatically. In `@cli/reins/src/lib/detection.ts`: - Around line 135-147: The function checkWorkflowConfigForPattern currently calls pattern.test(content) across multiple files but fails to reset RegExp.lastIndex when the caller supplies a global or sticky regex; update the loop in checkWorkflowConfigForPattern so that before each test you either reset pattern.lastIndex = 0 or use a fresh non-global clone of the caller's regex (e.g., new RegExp(pattern.source, pattern.flags.replace(/[gy]/g, ''))), ensuring pattern.test(content) behaves deterministically for each file. In `@README.md`: - Around line 272-280: Replace the incorrect heading "Self-apply: 19/22" with "Self-audit: 19/22" in the README; locate the markdown header string (the line starting with "## Self-apply") and update it to "## Self-audit" (and update any nearby occurrences/anchors if present) so the section accurately describes the repository audit score. In `@skill/Reins/Workflows/Audit.md`: - Around line 62-77: The Audit.md rubric is out of sync with the actual scorer in cli/reins/src/lib/audit/scoring.ts: update the rubric to match the scorer (or vice-versa if you intend to change behavior) by removing or correcting the Repository Knowledge entry for product-specs and the Agent Legibility entry for structured command output (these points are not currently awarded by scoring.ts), and explicitly state that cli/reins is the single source of truth for readiness scoring and JSON outputs; also audit the nearby section referenced (lines ~90-97) for the same mismatches and align wording/checklists with the exact signals and conditions implemented in scoring.ts (e.g., conditional-context and tool-registry are currently the signals used). - Around line 21-31: The example values in the JSON snippet use UI/web frameworks but the frameworks_detected field should list orchestration/engine frameworks; update the frameworks_detected example to appropriate orchestration names (e.g., "cli/reins", "airflow", "dagster", "kubernetes") so consumers infer the correct JSON shape—edit the frameworks_detected entry in the Audit.md snippet to replace "next.js" / "tailwind" with orchestration/engine examples and keep the array format consistent. --- Outside diff comments: In `@package.json`: - Line 8: The "audit:self" npm script uses an incorrect path token "../..." — update the script value for "audit:self" to use "../.." so the command cd cli/reins && bun src/index.ts audit ../.. correctly targets the repository root; locate the "audit:self" entry in package.json and replace the "../..." portion with "../..". In `@README.md`: - Around line 193-212: The fenced code block that lists files (starting with "AGENTS.md", "ARCHITECTURE.md", etc.) is missing a language tag and triggers markdownlint MD040; change the opening fence from ``` to a tagged fence such as ```text (or ```plaintext) so the block is recognized as plain text and the linter stops complaining—update the README.md fenced block that contains AGENTS.md/ARCHITECTURE.md/WORKFLOW.md to use ```text. --- Nitpick comments: In `@cli/reins/src/index.test.ts`: - Around line 179-190: The tests need to be updated to exercise the new orchestration scoring and explicit cutover thresholds: add at least one fixture directory (e.g., tmpDir("audit-orch-2sig") and tmpDir("audit-orch-3sig")) that produces 2 vs 3 orchestration signals, call runCli(`audit ${dir}`) and assert the orchestration-related fields (for example result.scores.orchestration.score and result.scores.orchestration.frameworks_detected) to show the 2→3 signal difference, and update the "returns correct maturity levels at boundaries" test to include direct assertions that specific total_score values hit the maturity cutoffs (5 vs 6, 11 vs 12, 16 vs 17, 19 vs 20) by constructing inputs or fixtures that produce totals at those thresholds and asserting maturity_level and total_score accordingly; touch the tests that reference runCli, tmpDir, result.total_score, result.maturity_level, and result.scores.* to locate where to add these checks. In `@cli/reins/src/lib/commands/init.ts`: - Around line 115-122: The ORDER of steps in the const steps array in init.ts places "Review WORKFLOW.md — customize agent behavior and orchestration settings" after the "Run 'reins audit .'" step; move the WORKFLOW.md entry earlier (for example, immediately before "Run 'reins audit .'" or after "Review risk-policy.json") so users customize workflow settings before running the initial audit, updating the steps array accordingly.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID:
61dccbfe-0d9a-4578-aae3-1b064180257d📒 Files selected for processing (22)
README.mdcli/reins/README.mdcli/reins/package.jsoncli/reins/src/index.test.tscli/reins/src/lib/audit/context.tscli/reins/src/lib/audit/scoring.tscli/reins/src/lib/commands/doctor.tscli/reins/src/lib/commands/evolve.tscli/reins/src/lib/commands/init.tscli/reins/src/lib/detection.tscli/reins/src/lib/templates.tscli/reins/src/lib/types.tsdocs/design-docs/ecosystem-positioning.mddocs/design-docs/index.mddocs/design-docs/symphony-integration.mdpackage.jsonskill/Reins/HarnessMethodology.mdskill/Reins/SKILL.mdskill/Reins/Workflows/Audit.mdskill/Reins/Workflows/Doctor.mdskill/Reins/Workflows/Evolve.mdskill/Reins/Workflows/Scaffold.md
… doctor with audit - Copy ctx.frameworksDetected to result.frameworks_detected in runAudit() - Doctor: add .codex/WORKFLOW.md to workflow config check - Doctor: require non-empty skills directories (match audit detection) - Doctor: include workflow-based merge protection (match audit detection) - Detection: strip /g and /y flags from caller-supplied RegExp to prevent statefulness - Evolve: mark workflow config step as automated (init scaffolds it) - Evolve: add WORKFLOW.md and SPEC.md to required artifacts for backfill - README: rename "Self-apply" to "Self-audit" - Audit.md: fix frameworks_detected example values and align rubric with scoring.ts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
cli/reins/src/lib/commands/evolve.ts (1)
1-1:⚠️ Potential issue | 🟡 MinorFix formatting to pass CI.
The pipeline indicates the formatter would apply changes. Run your project's formatter (e.g.,
bun run formator equivalent) before merging.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@cli/reins/src/lib/commands/evolve.ts` at line 1, The file contains formatting discrepancies (e.g., the import line `import { existsSync } from "node:fs";`) that fail CI; run the project's formatter (e.g., `bun run format` or your repo's configured formatter) and commit the resulting changes so the import and surrounding file match the project's style rules and CI passes.README.md (1)
193-212:⚠️ Potential issue | 🟡 MinorAdd language specifier to fenced code block.
The project structure code block is missing a language identifier. Adding one improves rendering and accessibility.
Suggested fix
-``` +```text AGENTS.md # Concise map (~100 lines) for agents🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@README.md` around lines 193 - 212, The fenced project-structure code block in README.md lacks a language specifier; update the opening fence for that block (the triple backticks immediately before the AGENTS.md listing) to include a language token such as text (e.g., change ``` to ```text) so the block renders/accessibility improve; locate the code block containing "AGENTS.md # Concise map (~100 lines) for agents" and modify its opening fence accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@skill/Reins/Workflows/Audit.md`:
- Around line 62-77: Update the rubric entry titled "Conditional context
engineering" to match the actual detection logic: change the description from
"3+ hierarchical AGENTS.md files" to "3+ hierarchical context files: AGENTS.md
and/or CLAUDE.md (or glob-based rule files)" so scoring reflects that cli/reins
counts both AGENTS.md and CLAUDE.md; leave the "Declared tool registry" and
"Orchestration readiness" criteria unchanged as they already match the code.
---
Outside diff comments:
In `@cli/reins/src/lib/commands/evolve.ts`:
- Line 1: The file contains formatting discrepancies (e.g., the import line
`import { existsSync } from "node:fs";`) that fail CI; run the project's
formatter (e.g., `bun run format` or your repo's configured formatter) and
commit the resulting changes so the import and surrounding file match the
project's style rules and CI passes.
In `@README.md`:
- Around line 193-212: The fenced project-structure code block in README.md
lacks a language specifier; update the opening fence for that block (the triple
backticks immediately before the AGENTS.md listing) to include a language token
such as text (e.g., change ``` to ```text) so the block renders/accessibility
improve; locate the code block containing "AGENTS.md # Concise map (~100 lines)
for agents" and modify its opening fence accordingly.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 0fd03224-30c6-4a39-8688-7edf61e106c5
📒 Files selected for processing (6)
README.mdcli/reins/src/lib/commands/audit.tscli/reins/src/lib/commands/doctor.tscli/reins/src/lib/commands/evolve.tscli/reins/src/lib/detection.tsskill/Reins/Workflows/Audit.md
The scoring logic counts both AGENTS.md and CLAUDE.md when evaluating hierarchical context engineering. Update the Audit.md rubric to match. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Extract 4 helpers from buildAuditRuntimeContext to reduce cognitive complexity from 26 to under 15 (biome lint) - Apply biome formatting fixes across scoring.ts, doctor.ts, evolve.ts - Fix evolve.test.ts: update max_score 21→22, agent_workflow max 4→5, add missing frameworks_detected field (TypeScript error) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
🧹 Nitpick comments (1)
cli/reins/src/lib/audit/context.ts (1)
78-86: Consider extendingdetectAgentCommandsto include.codex/skills.This function only checks
.claude/commands, but the PR introduces.codex/skillsas an equivalent skill directory. The separatedetectNonEmptySkillsDirectoryhandles this forhasSkillsDirectory, buthasAgentCommandsis used independently inscoreAgentWorkflowBlueprints.Consider whether this function should also check
.codex/skillsfor consistency, or if the current separation is intentional.🔧 Optional: extend detectAgentCommands
function detectAgentCommands(targetDir: string): boolean { - const commandsDir = join(targetDir, ".claude", "commands"); - if (!existsSync(commandsDir)) return false; - try { - return readdirSync(commandsDir).length > 0; - } catch { - return false; - } + const commandsDirs = [ + join(targetDir, ".claude", "commands"), + join(targetDir, ".codex", "skills"), + ]; + return commandsDirs.some((d) => { + if (!existsSync(d)) return false; + try { + return readdirSync(d).length > 0; + } catch { + return false; + } + }); }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@cli/reins/src/lib/audit/context.ts` around lines 78 - 86, detectAgentCommands currently only checks ".claude/commands" but the repo now also uses ".codex/skills"; update detectAgentCommands to treat ".codex/skills" as an equivalent by checking both directories (join(targetDir, ".claude", "commands") and join(targetDir, ".codex", "skills")) and returning true if either exists and is non-empty (same existsSync/readdirSync pattern and try/catch behavior). Keep the function signature detectAgentCommands(targetDir: string): boolean and ensure callers like hasAgentCommands and scoreAgentWorkflowBlueprints observe the expanded behavior; reuse the same error-handling semantics as the existing implementation.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@cli/reins/src/lib/audit/context.ts`:
- Around line 78-86: detectAgentCommands currently only checks
".claude/commands" but the repo now also uses ".codex/skills"; update
detectAgentCommands to treat ".codex/skills" as an equivalent by checking both
directories (join(targetDir, ".claude", "commands") and join(targetDir,
".codex", "skills")) and returning true if either exists and is non-empty (same
existsSync/readdirSync pattern and try/catch behavior). Keep the function
signature detectAgentCommands(targetDir: string): boolean and ensure callers
like hasAgentCommands and scoreAgentWorkflowBlueprints observe the expanded
behavior; reuse the same error-handling semantics as the existing
implementation.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 51ac6d0e-540a-4c04-98d8-de4de6ec9c75
📒 Files selected for processing (6)
cli/reins/src/lib/audit/context.tscli/reins/src/lib/audit/scoring.tscli/reins/src/lib/commands/doctor.tscli/reins/src/lib/commands/evolve.test.tscli/reins/src/lib/commands/evolve.tsskill/Reins/Workflows/Audit.md
Summary
Add Symphony integration support to Reins by detecting orchestration patterns generically (WORKFLOW.md, skills directories, workspace isolation, concurrency governance, merge protection). Expand
agent_workflowdimension from 0-4 to 0-5, increasing max score from 21 to 22 and recalibrating maturity levels (L0-L4).Changes
agent_workflow(awards point at 3+ of 5 signals)hasWorkflowConfig,hasSkillsDirectory,hasIsolationPolicy,hasConcurrencyLimits,hasMergeProtection,hasSpecDocument,frameworksDetected)reins initframeworks_detectedfield (informs which orchestration frameworks detected)Testing
bun testpasses (71/71)bun src/index.ts audit ../..self-score: 19/22 L3 (expected)Merge Readiness
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Enhancements
Documentation