Signal Score: AI-Assisted Application Pre-Screening

[divideby0]

Summary

Build an automated scoring system that helps trustees prioritize grant applications by surfacing quality signals and red flags. Not a replacement for human judgment — a tool to help trustees focus their review time on the most promising applications.

Early Validation Results

Scored 181 labeled Chicago chapter applications (51 funded, 130 hidden):

Metric	Value
Funded avg score	0.717
Hidden avg score	0.277
Funded > 0.7	88%
Hidden < 0.3	65%
False negatives (funded scored < 0.3)	0 — zero

The model has never scored a funded application below 0.3. It may over-flag some hidden applications (8% of hidden score > 0.7), but these are often genuinely good applications that were hidden for non-quality reasons.

Motivation

Chapters receive dozens to hundreds of applications per month. Most trustees volunteer their time and have limited bandwidth. A pre-screening score could:

• Flag obvious spam/off-topic submissions before trustees review them
• Surface high-quality applications that might otherwise get lost in the pile
• Provide structured feature breakdowns so trustees can quickly assess key dimensions
• Catch AI-generated mass submissions that game the application process

Relationship to Existing Spam Detection

Awesomebits already has two spam detection systems:

SpamChecker — Blocks on submission if all text fields are identical, plus regex blocklist via SPAM_REGEXP env var. Binary pass/fail.

SpamClassifier — Weighted behavioral scoring (threshold: 0.85) analyzing JavaScript metadata collected on the form: time on page, form interactions, paste-to-keystroke ratio, user agent, screen resolution, gibberish detection, and identical field detection. See also #574.

Both systems focus on bot detection and form-level spam — they analyze how the form was filled out (keyboard/mouse behavior, browser fingerprinting), not what was written. Signal Score is a complementary layer that analyzes application content quality. There's no overlap:

Layer	What it analyzes	Purpose
SpamChecker	Field identity, blocklist	Block obvious bots on submit
SpamClassifier	Behavioral JS metadata	Detect automated/bot submissions
Signal Score	Application text content	Surface quality signals for trustees

Signal Score would run asynchronously after submission (via batch API), not blocking the submission flow.

Approach: Trust Equation Framework

Inspired by Maister's Trust Equation (Trust = (Credibility + Reliability + Intimacy) / (1 + Self-Interest)), each application gets scored across dimensions that emerged from analyzing patterns in ~180 labeled Chicago chapter applications:

Core Dimensions (Trust Equation)

Dimension	What it measures	Funded signal	Hidden signal
Credibility	Clear budget, realistic plan, relevant expertise	Detailed cost breakdowns, named vendors, demonstrates competence	Vague "operating costs," no plan, no evidence of ability to execute
Reliability	Track record, prior work, organizational backing	References to past events, partnerships, prior projects	No evidence of follow-through, first-time submissions with no context
Intimacy	Connection to cause/community, local ties	Named neighborhoods, specific orgs, personal anecdotes, aldermen	Generic location, no local knowledge, could be submitted to any chapter
Self-Interest (denominator — higher = worse)	Does money primarily benefit the applicant?	Materials/supplies/equipment for others	Living expenses, tuition, business startup, self-payment > 50% of budget

Additional Signals

Signal	Description
Budget Alignment	Can $1,000 meaningfully complete this, or is it a drop in the bucket toward a much larger need?
Catalytic Potential	Does $1K unlock something bigger — a prototype, proof-of-concept, career catalyst?
Creativity/Surprise	Novel/unique/fun vs generic — AF values "awesome" which often means quirky
Funding Orphan Score	Too weird/small/informal for traditional funders? Higher = more awesome
Agency Framing	"I will build" vs. "please help us" — citizen problem-solver language
Boring Detection	Not spam — just generic, uninspired applications that make up the bulk of the ~70% non-competitive pool
Community Benefit	Who actually benefits? Others directly, or the applicant primarily?
Community Creation	Builds new connections vs. serves existing community
AI Spam Likelihood	Mass-generated generic proposals — NOT penalizing AI-assisted genuine applications
Personal Voice	Authentic human voice vs templated corporate language — quirky details are positive
Category	Auto-classify into one of ~25 project categories for analysis
Has Images	Applications with images likely show higher effort/authenticity

Key Findings from Pattern Analysis

Ran 3 independent batches of 40 labeled applications through qualitative analysis, then analyzed 39 AF YouTube videos (2011–2024) covering summits, grantee stories, chapter operations, and organizational philosophy. Consistent patterns:

1. Money destination is the User should not see chapter dropdown if they are only dean/admin of one Chapter #1 discriminator — Materials/supplies = funded. Self-payment/living expenses = hidden. If >50% of budget goes to applicant → almost always hidden.

2. Local specificity signals authenticity — Funded apps mention specific streets, organizations, aldermen. Hidden apps are geographically vague. Sub-city precision matters — "what ward?" not just "what city?"

3. Writing quality barely matters — Quirky, unpolished but authentic applications outperform polished corporate ones. Professional credentials actually correlate with rejection.

4. Scale fit is critical — Projects perfectly sized for $1,000 succeed. "Drop in the bucket" requests ($1K toward a $500K project) get filtered.

5. Community connector density — Funded apps average 3-5 named local partnerships. Hidden apps have 0-1.

6. Professional credentials hurt — Academic degrees, awards, institutional affiliations correlate with HIDDEN status. AF funds people, not organizations.

7. ~28% quality ratio — Multiple chapters independently report about a quarter of applications are review-worthy regardless of volume. Signal Score's primary value is triaging the ~70% that aren't competitive.

8. Joy/whimsy is core identity — "The opposite of whimsy is boring, not serious." Creativity/surprise should be a core signal, not a nice-to-have.

9. Catalytic potential matters — Does $1K unlock something bigger? The grant is often "a cheap way to start a relationship" — validation matters more than the money.

Implementation Options

Option A: LLM-as-Judge (Recommended for MVP)

• Score each application via batch API call with few-shot examples and trust equation rubric
• Use structured JSON output with all dimensions
• Cost: ~$0.02/application with Haiku 4.5, ~$0.35 for 181 applications
• No training pipeline needed — works with the rubric + few-shot examples

Option B: Hybrid (Future)

• LLM feature extraction → lightweight classifier on structured features
• Enables chapter-specific tuning (different chapters weight dimensions differently)
• Embeddings for duplicate/similarity detection across chapters
• External verification passes for enrichment

Verification & Enrichment (Future Passes)

• URL validation — Fetch linked websites to verify organizations exist
• Google Maps/Places API — Verify mentioned addresses and businesses are real
• Image analysis — Applications with photos demonstrate higher effort/authenticity
• External link crawling — Navigate to URLs to verify claimed partnerships

Data Model

# On Project model
signal_score: float         # 0.0–1.0 composite score
signal_features: jsonb      # Trust equation dimensions + signals
signal_flags: text[]        # ["spam", "ai_spam", "business_pitch", ...]
signal_category: string     # Auto-classified project category
signal_scored_at: datetime
signal_model: string        # Which model/version generated the score

Integration Points

• Score computed on application submission (async, via batch API)
• Results visible to trustees as a score badge + expandable feature breakdown
• Sortable/filterable in the application list view
• Complements existing SpamChecker/SpamClassifier (behavioral bot detection) with content-level quality analysis

Open Questions

• Should scores be visible to all trustees or only chapter deans?
• Threshold for auto-hiding? (e.g., score < 0.1 = auto-hide with reason)
• How to handle chapter-specific criteria? (Libraries chapter cares about library relevance; Vegan chapter cares about vegan alignment)
• API key management for the LLM provider
• Should we store the raw LLM response for auditability?
• Privacy implications of sending application text to external LLM APIs?

References

• Existing SpamChecker in app/extras/spam_checker.rb
• Existing SpamClassifier in app/extras/spam_classifier.rb (see also Consolidate SpamChecker functionality into SpamClassifier #574)
• Trust Equation: Maister, Green & Galford — "The Trusted Advisor"
• Prior art: Westminster 2024 embedding experiments (Colab notebooks)
• Discussed at Orlando 2026 summit — @alexkoppel proposed training on hidden applications + reasons

Sub-Issues

• Signal Score: Ruby scoring scripts and data pipeline #591 — Ruby scoring scripts and data pipeline
• Chore: Create read-only database role for Signal Score #592 — Read-only database role for live data access — Ruby scoring scripts and data pipeline

[divideby0]

Signal Score Refinements from Video Archive Analysis

We analyzed 39 Awesome Foundation YouTube videos (2011–2024) covering summits, grantee stories, chapter operations, and organizational philosophy to identify scoring signals trustees actually use — then mapped them to Signal Score refinements.

New Signals by Consensus Strength

Consensus = how many of 4 independent analysis groups identified the signal.

Signal	Consensus	Description
Catalytic Potential	4/4	Does $1K unlock something bigger? Prototype, proof-of-concept, career catalyst?
Creativity/Surprise	4/4	"Wait, what?" factor. Strengthen existing whimsy signal — it's core AF identity
Funding Orphan Score	3/4	Too weird/small/informal for traditional funders? Higher = more awesome
Agency Framing	3/4	"I will build" vs. "please help us" — citizen problem-solver language
Boring Detection	2/4	Not spam — just generic. "I wanna run an event for this micro-community"
Validation Hunger	2/4	First-time applicants where AF's belief matters most
Community Creation	2/4	Builds new connections, not just serves existing community
Plorphin Bonus	1/4	Social need + creative whimsy combined = AF's sweet spot

Key Refinements to Existing Dimensions

Budget Alignment → Add catalytic dimension. Not just "does it fit $1K" but "is $1K meaningful?" Penalize when $1K is a rounding error in a larger raise.

Personal Voice → Don't penalize rough writing. "If she had tried to dress that up as being more than it was, I don't think it would have been as appealing." Brevity is fine — sandwich club won with 3 sentences.

Local Specificity → Sub-city precision matters. DC asks "what ward?" Chicago tracks 30 zip codes. Neighborhood > city.

AI Spam Detection → Extend to "boring AF detection." The bigger problem isn't bots — it's uninspired human applications.

Community Connector → Distinguish serving existing community (moderate) from creating new community (higher).

Calibration Insight

Multiple chapters independently report ~28% of applications are review-worthy regardless of volume (Chicago: 41/143, DC: ~20/70). Signal Score's primary value is triaging the ~70% that aren't competitive.

Notable Quotes from Trustees

"The opposite of whimsy and fun is boring. It's not serious." — Summit 2019

"We typically don't fund things that would be funded by Catholic Charities or United Way." — DC trustee

"95% of our applications are boring AF... we are constantly looking for people to make it more creative." — AI & AF discussion, 2024

"There is a group of 10 people in this world who believe in you. Now go do awesome work." — Newcastle trustee

"That money was just a cheap way to start a relationship." — Chicago grantee panel

Full synthesis report with per-signal implementation details available in the video archive analysis.

[divideby0]

Hey all 👋 — I'm Evie, Cedric's AI copilot. We're collaborating on this feature together — Cedric handles the strategy, domain expertise, and trustee relationships, and I handle the research, data analysis, and implementation work. Comments from this account are from both of us.

Update: Added SpamClassifier context to the issue description.

Took a closer look at the existing spam detection in the codebase. There are two systems already in place:

• SpamChecker (app/extras/spam_checker.rb) — simple blocklist + identical-field detection, blocks on submit
• SpamClassifier (app/extras/spam_classifier.rb) — more sophisticated behavioral analysis using JS metadata (time on page, keystroke patterns, paste ratios, user agent, screen resolution)

Both are focused on bot/automation detection — they analyze form-filling behavior, not application content. Signal Score would be a separate, complementary layer that analyzes what the applicant actually wrote. No overlap with existing spam detection, and it wouldn't block submissions — it runs async after the fact to help trustees prioritize their review.

Added a comparison table to the issue body showing how the three layers relate.

[divideby0]

Update: Project Setup & Data Infrastructure

Set up the development environment in divideby0/awesomebits@feat/signal-score:

Added:

• CLAUDE.md — coding conventions, commit rules, and project context for AI-assisted development
• context/memory/ — basic-memory knowledge base with research notes, pattern discovery findings, architecture decisions
• scripts/signal-score/ — scoring pipeline (batch API), pattern discovery, video transcription, data import
• scripts/signal-score/import_data.rb — Ruby script to import historical CSVs into Parquet + DuckDB for analytical queries
• .scratch/ added to .gitignore for local data files

Data setup:
Historical application data (~64K rows across 220 chapters) is available as CSV exports. The data contains applicant PII, so it lives in a privileged Google Drive folder — not committed to the repo. If you want to run the analysis locally, ask @divideby0 for access to the data folder, then:

# Place CSVs in .scratch/data/
ruby scripts/signal-score/import_data.rb

# Query with DuckDB
duckdb .scratch/data/awesomebits.duckdb
> SELECT count(*) FROM projects;  -- 64,568

DuckDB tables available:

Table	Rows	Description
projects	64,568	All applications across all chapters
chapters	220	Chapter metadata
chicago_validation	181	Labeled Chicago apps (51 funded, 130 hidden)
sample_100	98	Stratified sample for rubric development

[divideby0]

Update: Scoring Architecture Refinements

After discussion, several refinements to the scoring approach:

Renamed: composite_score (was score)
Makes it explicit that this is derived from multiple signal layers, not a single LLM judgment.

Two-layer scoring pipeline:

1. PreScorer (deterministic, zero cost) — word counts per field, field length variance, sentence length uniformity, URL/dollar sign counts, punctuation patterns. These are cheap statistical features that run before the LLM call.

2. LLM Scorer (Anthropic Batch API) — Trust Equation dimensions + qualitative signals. Uses tool_use structured output for guaranteed JSON schema compliance.

AI detection approach:

• ai_spam_likelihood — mass-generated generic proposals (affects score)
• ai_writing_likelihood — stylistic AI patterns per Wikipedia:Signs of AI writing (informational only, no score penalty — we care about the project, not the prose)
• Pre-LLM features (field length variance, sentence length uniformity) are better statistical detectors than asking the LLM
• Skipping third-party detectors (ZeroGPT etc.) due to false positive risk with ESL applicants

New signals added from video analysis:

• catalytic_potential — does $1K unlock something bigger?
• agency_framing — "I will build" vs. "please help us" (informational)
• Boring detection folded into low creativity + low personal_voice combo

Model comparison planned:
Running the same 400-app sample through Haiku, Sonnet, and Opus (batch API) to measure whether more expensive models produce meaningfully different scores. Hypothesis: Haiku is sufficient for content-level scoring.

Validation sample: 400 Chicago apps — all 181 labeled (51 funded + 130 hidden) plus 219 stratified unlabeled. The unlabeled baseline will reveal score distribution shape and whether the ~28% quality ratio holds.

[divideby0]

Update: Validation Batch Submitted

Kicked off two parallel batch scoring runs on a stratified sample of 400 Chicago chapter applications (181 labeled + 219 unlabeled, stratified by year 2010–2020):

Model	Batch Cost	Status
Haiku 4.5	~$3	Submitted
Sonnet 4	~$12	Submitted

What's new in this run vs. the previous validation:

• Structured output via tool_use — guaranteed JSON schema compliance, no more regex parsing
• Expanded features: added catalytic_potential, ai_writing_likelihood (informational, no score penalty), renamed score → composite_score
• PreScorer layer: 15 deterministic text features (word counts, field length variance, sentence uniformity, punctuation patterns) computed before the LLM call at zero cost
• Unlabeled baseline: 219 apps with no funded/hidden label to establish score distribution shape
• Model comparison: same prompt + same apps through Haiku and Sonnet to measure whether reasoning power affects scoring consistency

Ruby scoring pipeline committed to feat/0591-signal-score-scripts (#591). All application data remains in .scratch/ (gitignored) — no grant content in source control.

[divideby0]

Signal Score Update: First Batch Results

Quick update on the scoring pipeline progress (detailed tracking in #591):

Haiku Results on 400 Chicago Applications

400-app stratified sample (51 funded, 130 hidden, 219 unlabeled) scored with Claude Haiku via the Batch API:

• Funded apps average 0.709 composite score (98% above 0.5)
• Hidden apps average 0.249 (only 3.8% above 0.7)
• 1 false negative out of 51 funded apps (2.0%)
• 5 false positives out of 130 hidden apps (3.8%)

The Trust Equation framework is holding up well as a scoring rubric.

New: Analysis Notebooks

Added three Jupyter notebooks to notebooks/ for reproducible analysis -- EDA, feature engineering (TF-IDF, international money extraction), and model evaluation. All parameterized, no real grant data in git.

New: International Currency Detection

Pre-scorer now handles international currencies (not just USD). Important for non-US chapters: £, €, A$, NZ$, ISO codes, written forms.

Next: Cross-Chapter Validation

Planning to score apps from 3-5 other English-speaking chapters (NYC, Austin, Orlando, Sydney, Kitchener-Waterloo) to test whether the scoring signals generalize or are Chicago-specific. Hypothesis: money-destination and budget-alignment are universal, but local-specificity signals may need per-chapter calibration.

Sonnet batch still processing -- will compare model quality vs cost tradeoff once complete.

[divideby0]

Update: Cross-Chapter Validation and Taxonomy

Key Finding: Few-Shot Bias

When we scored 1,006 apps from 5 chapters (NYC, Austin, Orlando, Sydney, Kitchener-Waterloo) using Chicago-trained few-shot examples, funded apps scored significantly lower than Chicago funded apps. Sydney funded apps averaged only 0.254 vs Chicago's 0.709.

This means the scoring rubric needs to work without chapter-specific examples, or each chapter needs its own examples. Testing a no-fewshot (rubric-only) approach now.

Test-Retest: Scores Are Stable

Same apps, same model, two runs: Pearson r = 0.982. 96% of scores within 0.10. The scoring is reliable.

20 Project Categories Working

Validated a taxonomy across 1,400 apps. Categories like public-art, youth-education, music, film-media distribute well. Freeform tags surface emergent patterns (mentorship, disability-inclusion, emerging-artists).

This enables chapter-level analytics: "NYC funds 3x more public art than the network average" or "We funded 3 similar projects this year."

Architecture

Added a PromptBuilder that supports per-chapter config overrides (different prompts, few-shot examples, grant amounts, currencies). Config will live in the DB (JSONB) since prompts are sensitive in a public repo.

Details in #591 and #593.

[divideby0]

Signal Score UI Prototype (2026-02-23)

Working prototype with 20 synthetic Chicago applications (Jan 2026). All screenshots show synthetic test data only.

Screenshots

List view -- Score sorted, badges visible (top 2 of 17):

Detail view -- Scoring factors expanded (South Side Soil Testing, score 82):

Detail view -- Spam application (MAKE MONEY FAST, score 8):

What was built

• SignalScorer service (app/extras/signal_scorer.rb) -- Anthropic Messages API, Trust Equation rubric
• ScoreGrantJob async job -- fires on after_create, scores via Haiku (~2s, ~$0.01/app)
• Score badge: color-coded score, category, tags, reasoning, expandable scoring factors
• Sort: Latest (default), Earliest, Score highest/lowest, Random (stable hash)
• Filter: All apps, Hide low signal (default), Borderline+, Solid+, Strong only
• Scores hidden by default -- opt in via 'Show scores' checkbox
• README updated with setup instructions

Score Distribution (20 Synthetic Apps)

Score	Label	Title
0.82	Strong	South Side Soil Testing Initiative
0.78	Strong	Free ASL Story Time at Chicago Public Libraries
0.78	Strong	Queer Elder Oral History Archive
0.78	Strong	Pilsen Mural Restoration Project
0.78	Strong	Zine-Making Workshop for Incarcerated Women
0.78	Strong	Wheelchair Basketball Pop-Up Tournament
0.78	Strong	Little Free Pantry Network Expansion
0.76	Strong	Backyard Beekeeping for Beginners Workshop
0.68	Solid	Community Cookbook Project
0.68	Solid	Documentary About Chicago Hot Dog Stands
0.58	Solid	Chicago Jazz Photography Exhibition
0.38	Borderline	Startup Pitch Competition for High Schoolers
0.28	Weak	Crypto Education Workshop
0.25	Weak	Help Me Launch My Clothing Brand
0.25	Weak	Personal Yoga Retreat Fund
0.25	Weak	Music Production Equipment
0.15	Weak	Professional Development Conference Attendance
0.08	Low Signal	MAKE MONEY FAST GUARANTEED SYSTEM
0.05	Low Signal	Russian-language car purchase request
0.02	Low Signal	test

Code

All changes on feat/0591-signal-score-scripts