Skip to content

test-all.mjs scans untracked files, causing false positives + silent misses #184

@matkatmusic

Description

@matkatmusic

Problem

The personal-data-leak check (test-all.mjs:~155) and absolute-path check (~205) both use bare grep -rn ., which walks the entire working directory and only excludes a handful of hardcoded paths (node_modules, .git/, go.sum). This produces two opposite-sign bugs:

Bug 1: False positives on untracked files

Any local-only file containing `/Users/` or one of the leak patterns triggers a test failure even though git would never ship the file. Common triggers:

  • Output from `/octo:debate` and similar AI tools (debates/, debate-output/)
  • Local plan drafts (`plans/`)
  • Personal scratch directories (`Todos/`, `scratch/`, `output/`)
  • IDE workspace files

I hit this with 6 failures from a debate/ session where the AIs echoed back absolute paths to my local repo. The files were untracked — they could never have reached a commit — but `grep -rn` didn't know that.

Bug 2: Silent misses on tracked files

The leak check uses `--include="*.{md,yml,html,mjs,sh,go,json}"` which relies on shell brace expansion. Apparently this isn't matching the i18n README files added in v1.3.0 (`README.es.md`, `README.ja.md`, `README.ko-KR.md`, `README.pt-BR.md`), because switching to `git grep` surfaces 36+ matches in those files that were going completely unscanned. They're all legitimate (they credit Santiago as the maintainer), but the test should have been seeing them all along to confirm the `allowedFiles` filter actually applies.

Fix

Replace `grep -rn` with `git grep -n` in both checks. `git grep` scans only files tracked by the index, so:

  • ✅ Untracked files can never false-positive
  • ✅ `.gitignore` is honored automatically
  • ✅ The manual `| grep -v node_modules | grep -v .git/ | grep -v go.sum` pipeline becomes dead code
  • ✅ Brace expansion is replaced with native pathspec syntax that actually works
  • ✅ The i18n READMEs get scanned for the first time

Side effect of the more thorough scanning: the previously-hidden matches in v1.3.0 community files (`CODE_OF_CONDUCT.md`, `SECURITY.md`, `SUPPORT.md`, `GOVERNANCE.md`, dashboard credits) need to be added to `allowedFiles` since they all legitimately credit the maintainer.

Verification

After the fix:

  • Plant `// /Users/santifer/test` in tracked `doctor.mjs` → still fails ✅
  • Plant same string in untracked `debates/test/test.md` → no false positive ✅
  • `node test-all.mjs --quick` → 63 passed, 0 failed, 0 warnings (was 6 failures + 0 warnings on the same working tree)

PR coming.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions