Signal Score: Ruby scoring scripts and data pipeline

[divideby0]

Port the Signal Score analysis tooling to Ruby to align with the existing Rails codebase.

Context

Initial prototyping for Signal Score (#590) was done in Bun/TypeScript. Now that the approach is validated, we should rewrite the scoring pipeline in Ruby so it can eventually become a Rails service object (app/extras/signal_scorer.rb) running via ActiveJob.

Scripts to Port

• score_grants.rb — Batch scoring via Anthropic API with Trust Equation rubric
• discover_patterns.rb — Qualitative pattern analysis across labeled applications
• import_data.rb — CSV → Parquet → DuckDB setup (already Ruby ✅)
• export_sample.rb — Stratified sample generation for validation

Dependencies

• anthropic gem — Batch API for scoring
• duckdb gem — Analytical queries on historical data
• dotenv-rails — Already in Gemfile, use .env for API keys

Data Setup

Historical application data (~64K rows) lives in a privileged Google Drive folder. Contact @divideby0 for access. Place CSVs in .scratch/data/ and run import_data.rb.

Refs: #590

[divideby0]

Progress Update: Notebooks, Pre-Scorer Features, and Haiku Batch Results

New: Analysis Notebooks (notebooks/)

Three Jupyter notebooks added for reproducible analysis, parameterized via AWESOMEBITS_DB env var (no real grant data in git):

• 01_eda.ipynb -- Corpus stats, chapter distributions, label rates, temporal trends, text field analysis across 64,568 applications
• 02_feature_engineering.ipynb -- Pre-scorer feature distributions, TF-IDF analysis (corpus-wide IDF table), international money extraction, feature correlations
• 03_model_evaluation.ipynb -- Batch result analysis, model comparison (Haiku vs Sonnet), test-retest reliability framework, cross-chapter validation

New Pre-Scorer Features

• money_mention_count -- International currency extraction (replaces US-only dollar_sign_count). Handles symbols ($, £, €, ¥, ₹), prefixed symbols (A$, NZ$, CA$), ISO codes (USD, GBP, EUR), and written forms (dollars, pounds, euros).
• TF-IDF features (tfidf_mean, tfidf_max, tfidf_unique_terms) -- Loaded from a static IDF table computed across the full corpus. Captures vocabulary distinctiveness without runtime recomputation.

Haiku Batch Results (400 apps)

Batch msgbatch_01Cgsu53oWAbAcGGL6TwWxnV completed (400/400 succeeded, 0 errors):

Label	n	Mean Score	Median	>0.5	>0.7
Funded	51	0.709	0.720	98.0%	76.5%
Hidden	130	0.249	0.250	10.8%	3.8%
Unlabeled	219	0.449	0.420	46.1%	12.8%

False negatives (funded < 0.5): 1/51 (2.0%)
False positives (hidden > 0.7): 5/130 (3.8%)

Separation is strong. Only 1 funded app scored below 0.5, and only 5 hidden apps scored above 0.7.

Sonnet Batch

msgbatch_0182PJyuUvpV32biR1py28js still processing (400 apps). Will compare head-to-head once complete.

Planned Next

1. Cross-chapter validation -- run same pipeline on NYC, Austin, Orlando, and 1-2 international chapters (Sydney, Kitchener-Waterloo) to test whether scoring signals are locality-specific
2. Test-retest reliability -- resubmit same Haiku JSONL to measure score variance across runs
3. Static IDF table export from notebook 02 for Ruby pre-scorer integration

Branch: feat/0591-signal-score-scripts on divideby0/awesomebits

[divideby0]

Update: Cross-Chapter Results, Reliability, Taxonomy, and Refactoring

Test-Retest Reliability (Haiku)

Same 400 Chicago apps, same prompt, two separate batch runs:

• Pearson r = 0.982 (near-perfect correlation)
• 80.5% of scores within 0.05, 96% within 0.10
• Threshold agreement: 95-96% at all cutoffs (0.3, 0.5, 0.7)
• Median absolute difference: 0.000 (most scores identical)
• Max drift: 0.20 (single outlier)

Verdict: scores are highly stable despite temperature=1.0.

Cross-Chapter Scoring (with Chicago few-shot examples)

Scored 1,006 apps across 5 chapters using Chicago-trained few-shot examples:

Chapter	Funded Mean	Hidden Mean	Funded >0.5
NYC	0.409	0.173	28.6%
Austin	0.465	0.244	40.0%
Orlando	0.433	0.287	36.7%
Sydney	0.254	0.158	6.7%
KW	0.349	0.189	21.1%
Chicago (baseline)	0.709	0.249	98.0%

Finding: Few-shot examples create chapter bias. Chicago-trained examples anchor the model to Chicago-style applications. Sydney and KW funded apps score dramatically lower. Hidden apps are consistently low everywhere (good), but funded app scores are depressed.

Mitigation in progress: Submitted no-fewshot batches (rubric-only, no examples) for both Chicago and cross-chapter. This will tell us if the rubric alone generalizes better.

Taxonomy (20 Categories)

Categories validated on 1,400+ apps. Top primary categories for Chicago:

public-art (63), literary (56), youth-education (48), film-media (34), music (29), civic-social-justice (25), theater (24), festival-event (20)

Top freeform tags (emergent): mentorship, small-business, emerging-artists, disability-inclusion, interactive-installation, self-publishing

Code Changes

• PromptBuilder (scripts/signal-score/prompt_builder.rb): Extracts prompt assembly, supports chapter-specific config overrides, configurable grant amount/currency
• AnthropicCache (scripts/signal-score/anthropic_cache.rb): File-based passthrough cache, skips re-scoring cached apps
• score_grants.rb refactored: Uses PromptBuilder and cache, new flags: --chapter, --no-fewshot
• categories.json enriched: slug + name + description per category
• Issue Signal Score: test infrastructure, config model, and response cache #593 created: Test infrastructure, config model, response cache

Pending Batches

1. Sonnet Chicago (400) -- model comparison
2. Cross-chapter no-fewshot (1,006) -- rubric-only generalization test
3. Chicago no-fewshot (400) -- apples-to-apples comparison

Branch: feat/0591-signal-score-scripts

[divideby0]

Milestone: Few-Shot Bias Confirmed and Fixed

The Problem

Chicago few-shot examples were anchoring the model to Chicago-style applications. Sydney funded apps averaged 0.254 (should be ~0.7).

The Fix: Rubric-Only (No Few-Shot) Scoring

Resubmitted the same 1,006 cross-chapter apps with only the scoring rubric (no examples). Results:

Chapter	Funded (few-shot)	Funded (no-fewshot)	Hidden (few-shot)	Hidden (no-fewshot)	Separation (no-FS)
NYC	0.409	0.698	0.173	0.297	0.401
Austin	0.465	0.700	0.244	0.431	0.269
Orlando	0.433	0.688	0.287	0.487	0.201
Sydney	0.254	0.713	0.158	0.385	0.328
KW	0.349	0.686	0.189	0.451	0.235
Chicago	0.709	0.734	0.249	0.332	0.402

Analysis

1. Funded scores normalized across all chapters -- all chapters now cluster around 0.69-0.73 for funded apps. The rubric alone generalizes well.

2. Tradeoff: hidden scores inflated -- without examples of what "bad" looks like, the model is more generous with hidden apps (Chicago hidden went from 0.249 to 0.332, Orlando from 0.287 to 0.487).

3. Separation still strong for most chapters -- NYC (0.401) and Chicago (0.402) have excellent funded/hidden separation. Sydney (0.328) improved dramatically from 0.096.

4. Orlando is the hardest chapter -- only 0.201 separation. Orlando has 178 hidden apps (heavy use of hide feature) which may include borderline apps that are genuinely decent.

Recommendation

Use rubric-only as the default, with optional chapter-specific few-shot examples loaded from signal_score_configs.prompt_overrides.few_shot_project_ids. Chapters that have curated their own examples can opt in; others get the universal rubric.

The threshold may need to be chapter-configurable too. Chicago's 0.5 threshold captures 96% of funded apps with rubric-only. Orlando might need 0.4.

Also Committed

• signal_score_configs migration (JSONB, per-chapter config)
• SignalScoreConfig model (deep-merge resolution)
• SignalScorer class (Rails entry point)
• Model spec with resolve/merge tests

[divideby0]

Batch Results: 3-Way Scoring Comparison (2026-02-23)

Five batches completed — Sonnet vs Haiku model comparison, plus three prompt variants (Chicago few-shot, no few-shot, universal few-shot) tested on both Chicago (400 apps) and cross-chapter (1,006 apps from 5 chapters).

Chicago 400 Results

Variant	Funded avg	Hidden avg	Gap	FN@0.5	FP@0.5
Haiku (Chicago few-shot)	0.709	0.249	0.460	2.0%	10.8%
Sonnet (Chicago few-shot)	0.734	0.257	0.477	2.0%	11.5%
Haiku (no few-shot)	0.734	0.332	0.402	3.9%	21.5%
Haiku (universal few-shot)	0.699	0.296	0.403	3.9%	14.6%

Sonnet marginally outperforms Haiku (r=0.886 correlation). At 50% cost premium, Haiku remains the better choice for pre-screening.

Cross-Chapter Bias Fix

Funded averages by chapter across prompt variants:

Chapter	Chicago-FS	No-FS	Universal-FS
Sydney	0.254	0.713	0.665
Kitchener-Waterloo	0.349	0.686	0.654
NYC	0.409	0.698	0.647
Orlando	0.433	0.688	0.625
Austin	0.465	0.700	0.662

Chicago few-shot examples were severely depressing non-Chicago scores. Removing or replacing them with universal examples lifts all chapters to comparable ranges (0.625-0.713).

The Tradeoff

Cross-chapter overall (190 funded, 335 hidden):

Variant	Funded avg	Hidden avg	Gap	FP@0.5
Chicago few-shot	0.382	0.251	0.131	6.6%
No few-shot	0.699	0.442	0.257	43.9%
Universal few-shot	0.652	0.401	0.251	36.1%

Fixing cross-chapter equity raises hidden scores too. FP rate jumps from 6.6% to 36-44%.

Recommendations

1. Universal few-shot is the sweet spot for multi-chapter deployment
2. Chapter-specific threshold tuning needed (not a universal 0.5 cutoff)
3. Pre-scorer features as second filter — word count, money detection, TF-IDF can tighten FP rate
4. Haiku over Sonnet — marginal improvement does not justify 2x cost for batch scoring

Batch IDs

• Sonnet Chicago: msgbatch_0182PJyuUvpV32biR1py28js
• Universal-FS cross-chapter: msgbatch_01TJxj5bKexj4q8bmTRJJvk1
• Universal-FS Chicago: msgbatch_01GVpdxb5fzXSgFX7enkRRbo
• No-FS cross-chapter: msgbatch_011ZiXdPrhtSTPugu9bftzPC
• No-FS Chicago: msgbatch_01GroXafg5HLEq9PKg3xTbXg

[avicaplan]

Great to see this coming together! I might humbly suggest that the Kitchener-Waterloo chapter is not an ideal one to test on – they have never really engaged with our wider global community, and don't seem to consistently use the awesomefoundation.org platform. Indeed, while they are still active their last posted grants are from 2022. I would suggest using instead the Ottawa chapter.