Signal Score: test infrastructure, config model, and response cache

[divideby0]

Overview

Build the test and config infrastructure needed before Signal Score can move into the Rails app.

Components

1. signal_score_configs table (JSONB)

create_table :signal_score_configs do |t|
  t.references :chapter, null: true, foreign_key: true  # null = global default
  t.jsonb :scoring_config, null: false, default: {}
  t.jsonb :prompt_overrides, null: false, default: {}
  t.jsonb :category_config, null: false, default: {}
  t.boolean :enabled, default: false
  t.timestamps
end
add_index :signal_score_configs, :chapter_id, unique: true

Config resolution: chapter-specific deep-merges over global default (chapter_id = NULL).

• scoring_config: model, temperature, few_shot_count, score_threshold, grant_amount, currency
• prompt_overrides: system_prompt (full override), rubric_addendum (appended), few_shot_project_ids (curated examples)
• category_config: weights (boost/dampen), disabled (irrelevant), custom (chapter-specific)

2. Anthropic response cache

File-based passthrough cache at .scratch/cache/anthropic/{hash}.json.

• Cache key: SHA256 of (model + system prompt + tool schema + user message)
• On hit: return cached response, no API call
• On miss: call API, write response, return
• Batch integration: batch builder skips cached, result parser writes to cache
• Test integration: seed cache with real responses for deterministic specs

3. Test infrastructure

Seed task: lib/tasks/signal_score_seed.rake -- loads real data from .scratch/data/awesomebits.duckdb. Fails if missing.

RSpec config: :signal_score_data tag, excluded by default.

SIGNAL_SCORE_DATA=1 bundle exec rspec --tag signal_score_data

Specs:

• spec/lib/signal_score/pre_scorer_spec.rb
• spec/lib/signal_score/prompt_builder_spec.rb
• spec/lib/signal_score/config_spec.rb
• spec/lib/signal_score/scorer_spec.rb
• spec/lib/signal_score/categorizer_spec.rb

4. PromptBuilder class

app/extras/signal_score/prompt_builder.rb -- assembles system prompt, few-shot examples, tool schemas. Loads config from DB, deep-merges chapter overrides.

5. Canonical categories

config/signal_score/categories.json (20 categories, git-tracked):

public-art, community-garden, youth-education, music, food-community, film-media, literary, dance, theater, environment, sports-recreation, festival-event, technology-stem, women-girls, health-wellness, children-families, civic-social-justice, cultural-heritage, immigrant-refugee, housing-homelessness

Plus 3-5 freeform tags per app (unbounded).

Notes

• Real data in .scratch/ (gitignored), never committed
• Prompts/rubrics in DB, not files (public repo, gameable if visible)
• Cache doubles as test fixtures
• Funded project text already public on awesomefoundation.org

Refs: #591, #590

[divideby0]

Status Update

Test infrastructure and config model are implemented and passing:

Implemented

• signal_score_configs table — JSONB config with global defaults (chapter_id = NULL) + chapter-specific overrides, deep-merge resolution
• SignalScoreConfig model — uniqueness validations, .resolve(chapter) with deep-merge
• Anthropic response cache — File-based SHA256-keyed passthrough cache at .scratch/cache/anthropic/
• PromptBuilder — System prompt + tool schema + few-shot assembly
• WebMock added for HTTP stubbing in tests
• PreScorer specs — deterministic text feature extraction

Test Coverage (34 specs total across all signal score components)

• spec/models/signal_score_config_spec.rb — association, uniqueness validations, .resolve deep-merge
• spec/lib/signal_score/pre_scorer_spec.rb — text feature extraction
• spec/extras/signal_scorer_spec.rb — full API integration (WebMock-stubbed)
• spec/jobs/score_grant_job_spec.rb — async job lifecycle

All passing. PR: incoming from feat/0591-signal-score-scripts

Refs: #591, #590