Feature/bench scorecard ci windows fixes

.agents/skills/design-contracts/SKILL.md

@@ -0,0 +1,116 @@
+---
+name: design-contracts
+description: Enforces smriti's three design contracts (observability, dry-run, versioning) when writing or modifying CLI command handlers or JSON output.
+---
+
+# smriti Design Contract Guardrails
+
+This skill activates whenever you are **adding or modifying a CLI command**,
+**changing JSON output**, **touching telemetry/logging code**, or **altering
+config defaults** in the smriti project.
+
+---
+
+## Contract 1 — Dry Run
+
+### Mutating commands MUST support `--dry-run`
+
+The following commands write to disk, the database, or the network. Every one of
+them **must** honour `--dry-run`:
+
+| Command      | Expected guard pattern                                                        |
+| ------------ | ----------------------------------------------------------------------------- |
+| `ingest`     | `const dryRun = hasFlag(args, "--dry-run");` then no DB/file writes when true |
+| `embed`      | same                                                                          |
+| `categorize` | same                                                                          |
+| `tag`        | same                                                                          |
+| `share`      | same                                                                          |
+| `sync`       | same                                                                          |
+| `context`    | already implemented — keep it                                                 |
+
+When `--dry-run` is active:
+
+- `stdout` must describe **what would happen** (e.g. `Would ingest N sessions`).
+- `stderr` must note what was skipped (`No changes were made (--dry-run)`).
+- Exit code follows normal success/error rules — dry-run is NOT an error.
+- If `--json` is also set, the output envelope must include
+  `"meta": { "dry_run": true }`.
+
+### Read-only commands MUST reject `--dry-run`
+
+These commands never mutate state. If they receive `--dry-run`, they must print
+a usage error and `process.exit(1)`:
+
+`search`, `recall`, `list`, `status`, `show`, `compare`, `projects`, `team`,
+`categories`
+
+---
+
+## Contract 2 — Observability / Telemetry
+
+### Never log user content
+
+The following are **forbidden** in any `console.log`, `console.error`, or
+log/audit output:
+
+- Message content (`.content`, `.text`, `.body`)
+- Query strings passed by the user
+- Memory text or embedding data
+- File paths provided by the user (as opposed to system-derived paths)
+
+✅ OK to log: command name, exit code, duration, session IDs, counts, smriti
+version.
+
+### Telemetry default must be OFF
+
+- `SMRITI_TELEMETRY` must default to `0`/`false`/`"off"` — never `1`.
+- Telemetry calls must be guarded: `if (telemetryEnabled) { ... }`.
+- Any new telemetry signal must be added to `smriti telemetry sample` output.
+
+---
+
+## Contract 3 — JSON & CLI Versioning
+
+### JSON output is a hard contract
+
+The standard output envelope is:
+
+```json
+{ "ok": true, "data": { ... }, "meta": { ... } }
+```
+
+Rules:
+
+- **Never remove a field** from `data` or `meta` — add `@deprecated` in a
+  comment instead.
+- **Never rename a field**.
+- **Never change a field's type** (e.g. string → number).
+- New fields in `data` or `meta` must be **optional**.
+- If you must replace a field: add the new one AND keep the old one with a
+  `_deprecated: true` sibling or comment.
+
+### CLI interface stability
+
+Once a command or flag has shipped:
+
+- **Command names**: frozen.
+- **Flag names**: frozen. You may add aliases (e.g. `--dry-run` → `-n`) but not
+  rename.
+- **Positional argument order**: frozen.
+- **Deprecated flags**: must keep working, must emit a `stderr` warning.
+
+---
+
+## Pre-Submission Checklist
+
+Before finishing any edit that touches `src/index.ts` or a command handler:
+
+- [ ] If command is mutating → `--dry-run` is supported and guarded
+- [ ] If command is read-only → `--dry-run` is rejected with a usage error
+- [ ] No user-supplied content appears in `console.log`/`console.error`
+- [ ] If JSON output changed → only fields were **added**, not
+      removed/renamed/retyped
+- [ ] If a new flag was added → it does not conflict with any existing flag name
+- [ ] Telemetry default remains off in `config.ts`
+
+If any item fails, fix it before proceeding.

.github/workflows/ci.yml

@@ -7,7 +7,31 @@ on:
     branches: [main, dev]
 
 jobs:
-  test:
+  test-pr:
+    if: github.event_name == 'pull_request'
+    name: Test (ubuntu-latest)
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          submodules: recursive
+
+      - name: Setup Bun
+        uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: latest
+
+      - name: Install dependencies
+        run: bun install
+
+      - name: Run tests
+        # Fast PR validation on Linux only.
+        run: bun test test/
+
+  test-merge:
+    if: github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/dev')
     name: Test (${{ matrix.os }})
     runs-on: ${{ matrix.os }}
     strategy:
@@ -30,7 +54,5 @@ jobs:
         run: bun install
 
       - name: Run tests
-        # We only run tests in the smriti/test directory.
-        # qmd/ tests are skipped here as they are part of the backbone submodule
-        # and may have heavy dependencies (like local LLMs) that the runner lacks.
+        # Full cross-platform test matrix for merge branches.
         run: bun test test/

.github/workflows/dev-draft-release.yml

@@ -0,0 +1,73 @@
+name: Dev Draft Release
+
+on:
+  workflow_run:
+    workflows: ["CI"]
+    types: [completed]
+
+jobs:
+  draft-release:
+    name: Create/Update Dev Draft Release
+    if: >
+      github.event.workflow_run.conclusion == 'success' &&
+      github.event.workflow_run.event == 'push' &&
+      github.event.workflow_run.head_branch == 'dev'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+
+    steps:
+      - name: Checkout dev commit
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.workflow_run.head_sha }}
+          fetch-depth: 0
+          submodules: recursive
+
+      - name: Compute dev tag
+        id: tag
+        run: |
+          BASE_VERSION=$(node -p "require('./package.json').version")
+          DEV_SUFFIX="dev.${{ github.event.workflow_run.run_number }}"
+          DEV_TAG="v${BASE_VERSION}-${DEV_SUFFIX}"
+          echo "base_version=${BASE_VERSION}" >> "$GITHUB_OUTPUT"
+          echo "dev_tag=${DEV_TAG}" >> "$GITHUB_OUTPUT"
+
+      - name: Remove previous dev draft releases
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const releases = await github.paginate(github.rest.repos.listReleases, {
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              per_page: 100,
+            });
+
+            for (const release of releases) {
+              const isDevTag = /-dev\.\d+$/.test(release.tag_name || "");
+              if (isDevTag && release.draft) {
+                await github.rest.repos.deleteRelease({
+                  owner: context.repo.owner,
+                  repo: context.repo.repo,
+                  release_id: release.id,
+                });
+              }
+            }
+
+      - name: Remove previous dev tags
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          for tag in $(git tag --list 'v*-dev.*'); do
+            git push origin ":refs/tags/${tag}" || true
+          done
+
+      - name: Create dev draft prerelease
+        uses: softprops/action-gh-release@v2
+        with:
+          tag_name: ${{ steps.tag.outputs.dev_tag }}
+          target_commitish: ${{ github.event.workflow_run.head_sha }}
+          name: Dev Draft ${{ steps.tag.outputs.dev_tag }}
+          generate_release_notes: true
+          draft: true
+          prerelease: true

.github/workflows/perf-bench.yml

@@ -0,0 +1,105 @@
+name: Perf Bench (Non-blocking)
+
+on:
+  pull_request:
+    branches: [main, dev]
+    paths:
+      - "src/**"
+      - "qmd/src/**"
+      - "scripts/bench-*.ts"
+      - "bench/**"
+      - ".github/workflows/perf-bench.yml"
+  push:
+    branches: [main, dev, "feature/**"]
+    paths:
+      - "src/**"
+      - "qmd/src/**"
+      - "scripts/bench-*.ts"
+      - "bench/**"
+      - ".github/workflows/perf-bench.yml"
+
+jobs:
+  bench:
+    name: Run ci-small benchmark
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      pull-requests: write
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          submodules: recursive
+
+      - name: Setup Bun
+        uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: latest
+
+      - name: Install dependencies
+        run: bun install
+
+      - name: Run benchmark (no-llm)
+        run: bun run scripts/bench-qmd.ts --profile ci-small --out bench/results/ci-small.json --no-llm
+
+      - name: Run repeated benchmark (ci-small)
+        run: bun run scripts/bench-qmd-repeat.ts --profiles ci-small --runs 3 --out bench/results/repeat-summary.json
+
+      - name: Compare against baseline (non-blocking)
+        run: bun run scripts/bench-compare.ts --baseline bench/baseline.ci-small.json --current bench/results/ci-small.json --threshold 0.2
+
+      - name: Generate scorecard markdown
+        run: bun run bench:scorecard > bench/results/scorecard.md
+
+      - name: Add scorecard to run summary
+        run: |
+          echo "## Benchmark Scorecard (ci-small)" >> "$GITHUB_STEP_SUMMARY"
+          echo "" >> "$GITHUB_STEP_SUMMARY"
+          cat bench/results/scorecard.md >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Upsert sticky PR comment
+        if: github.event_name == 'pull_request'
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const fs = require("fs");
+            const body = fs.readFileSync("bench/results/scorecard.md", "utf8");
+            const marker = "<!-- bench-scorecard -->";
+            const fullBody = `${marker}
+            ## Benchmark Scorecard (ci-small)
+
+            ${body}`;
+            const { owner, repo } = context.repo;
+            const issue_number = context.payload.pull_request.number;
+            const comments = await github.paginate(github.rest.issues.listComments, {
+              owner,
+              repo,
+              issue_number,
+              per_page: 100,
+            });
+            const existing = comments.find((c) => c.body && c.body.includes(marker));
+            if (existing) {
+              await github.rest.issues.updateComment({
+                owner,
+                repo,
+                comment_id: existing.id,
+                body: fullBody,
+              });
+            } else {
+              await github.rest.issues.createComment({
+                owner,
+                repo,
+                issue_number,
+                body: fullBody,
+              });
+            }
+
+      - name: Upload benchmark artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: bench-ci-small
+          path: |
+            bench/results/ci-small.json
+            bench/results/repeat-summary.json
+            bench/results/scorecard.md

.github/workflows/validate-design.yml

@@ -0,0 +1,36 @@
+name: Design Contracts
+
+on:
+  push:
+    branches: [main, dev, "feature/**"]
+    paths:
+      - "src/**"
+      - "scripts/validate-design.ts"
+      - "docs/DESIGN.md"
+  pull_request:
+    branches: [main, dev]
+    paths:
+      - "src/**"
+      - "scripts/validate-design.ts"
+      - "docs/DESIGN.md"
+
+jobs:
+  validate:
+    name: Validate Design Contracts
+    if: ${{ false }} # Temporarily disabled while validator rules are being aligned with current CLI behavior.
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Bun
+        uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: latest
+
+      - name: Install dependencies
+        run: bun install
+
+      - name: Run design contract validator
+        run: bun run scripts/validate-design.ts