feat: filter expired and stale job postings by posting date

[matkatmusic]

Closes #163.

Summary

Adds freshness filtering to scan and pipeline so months-old cached search results are discarded before they consume evaluation tokens. The existing Step 7.5 liveness check in scan.md only runs via Playwright (unavailable in batch mode / claude -p), so in batch workflows nothing currently enforces freshness at all. On a first-time scan with 58 results, ~25% turned out to be 2-4 year old LinkedIn postings or already-filled Greenhouse reqs that still showed apply buttons.

Approach

Extended the existing check-liveness.mjs rather than adding a new script. Detection is layered by cost:

1. LinkedIn URL ID heuristic — job IDs are sequential and leak posting year. Rejects 2y+ old LinkedIn URLs with zero network.
2. JSON-LD datePosted — embedded server-side by Greenhouse, Ashby, Lever, and most LinkedIn postings. Works through fetch() (no JS execution needed).
3. Inline "datePosted":"..." patterns — for minified embeds outside JSON-LD blocks.
4. Visible text patterns — Posted YYYY-MM-DD, Posted Aug 15, 2025, Posted N days/weeks/months ago.
5. Existing ?error=true redirect + body text patterns.

If JSON-LD returns a fresh datePosted, that short-circuits the "insufficient content" rejection rule — SPAs like Ashby have rich JSON-LD payloads but very little stripped bodyText, so the old check incorrectly flagged them expired.

Changes

check-liveness.mjs — +254 lines

• New --fetch-mode (HTTP-only, parallel-safe, for batch workers)
• New --json output (machine-consumable)
• New --classify output (just fresh|stale|expired|unverified)
• New extractPostingDate(), linkedinIdToYear(), classifyFreshness(), loadFreshnessConfig() — all exported for tests
• Existing Playwright path preserved and extended with the same date extraction

templates/portals.example.yml — new `freshness:` block:
```yaml
freshness:
max_age_days: 60 # Hard skip
warn_age_days: 30 # Evaluate but apply Red Flags penalty
linkedin_suspect: true # Treat LinkedIn cache snippets as unverified
require_date: false # Strict mode opt-in
```

modes/scan.md — Step 7.5 reworked to call check-liveness (Playwright or --fetch-mode). New skipped_stale status in scan-history.tsv. Output summary includes stale count.

modes/pipeline.md — New Step 2c freshness pre-filter. Stale URLs get a minimal SKIPPED_STALE report without an A-F eval (saves tokens on dead links). **Posted:** is now a required field in all report headers.

modes/_shared.md — Codifies the automatic Red Flags penalty for stale postings (-0.5 for stale, pipeline.md skips expired entirely). Adds ALWAYS rule and Tools table entry.

docs/ARCHITECTURE.md + docs/CUSTOMIZATION.md — Full freshness section with detection pipeline diagram and per-market tuning guidance.

Tests

New test-freshness.mjs — 40 pure unit tests (no network, no Playwright):

• JSON-LD extraction: top-level, @graph, multiple blocks, malformed handling, inline minified
• Visible text patterns: ISO, long-form, days/weeks/months ago, null inputs
• LinkedIn ID heuristic: 7 buckets + edge cases
• Freshness classification: all 4 states + boundaries at warn_age_days/max_age_days
• Config loader: default fallback + portals.yml override

Wired into test-all.mjs. Runs in well under a second.

End-to-end verification

Ran against a curated corpus from a real first-time scan I did earlier today:

Set	Result
9 known-stale URLs (5 old LinkedIn, 2 Greenhouse redirects, 1 Ashby 138d old, 1 Lever 404)	9/9 caught
1 known-fresh URL (Suno DAW engineer, 8d old, JSON-LD confirmed)	1/1 preserved

The fresh one is particularly important because naive content-length heuristics would reject Ashby SPAs — the JSON-LD short-circuit fixes that.

Open questions for review

1. I put freshness: in portals.yml because it feels like scanner behavior, but a case exists for config/profile.yml since different users in different markets might want different thresholds. Happy to move it.
2. The LinkedIn ID→year table needs periodic recalibration. I added a comment noting this but no automation yet. Could add a --calibrate-linkedin mode later.
3. fetch-mode deliberately doesn't execute JS, so fully client-side SPAs (like Apple's careers site) return uncertain when they have no server-rendered JSON-LD. Current behavior: classify as fresh (no date + require_date: false). Open to stricter default if you prefer.

Test plan

• node test-freshness.mjs — all 40 unit tests pass
• node test-all.mjs --quick — freshness suite green (pre-existing debate/ absolute path warnings unrelated)
• End-to-end: 9/9 stale caught, 1/1 fresh preserved against real URLs
• Verify Playwright path still works on active Greenhouse/Ashby/Lever URL
• Verify --fetch-mode parallel execution is safe (no shared browser state)
• Full scan + pipeline run on user's real portals.yml (deferred to post-merge)

Data contract

All changes land in the system layer. Zero touches to cv.md, config/profile.yml, data/*, or user's portals.yml. Users opt into freshness filtering by copying the block from portals.example.yml during their next update-system.mjs apply.

🤖 Generated with Claude Code