feat: Signal Score — AI-assisted application pre-screening#594
Open
divideby0 wants to merge 25 commits intoawesomefoundation:masterfrom
Open
feat: Signal Score — AI-assisted application pre-screening#594divideby0 wants to merge 25 commits intoawesomefoundation:masterfrom
divideby0 wants to merge 25 commits intoawesomefoundation:masterfrom
Conversation
Add CLAUDE.md, basic-memory context, data import script, .env.local.example, and gitignore updates. Refs: awesomefoundation#591
Non-secret configuration defaults for local dev. Personal secrets go in .env.local (see .env.local.example). Refs: awesomefoundation#591
README documents .env as standard location for secrets (AWS keys, etc). Add note that .env.local takes precedence for personal API keys. Refs: awesomefoundation#591
PreScorer for deterministic text features. ScoreGrants for Anthropic batch API with tool_use structured output. Supports haiku/sonnet/opus. Refs: awesomefoundation#591
- Add notebooks/01_eda.ipynb: corpus stats, chapter distributions, label rates, text field analysis - Add notebooks/02_feature_engineering.ipynb: pre-scorer features, TF-IDF analysis, money extraction, feature correlations - Add notebooks/03_model_evaluation.ipynb: batch results, model comparison, test-retest reliability, cross-chapter analysis - Add notebooks/helpers.py: shared data source, plotting, money extraction, label helpers - Add international money/currency extraction to pre_scorer.rb (supports symbols, prefixed symbols, ISO codes, written forms) - Add TF-IDF feature support to pre_scorer.rb (static IDF table computed from notebooks, loaded at runtime) - Replace dollar_sign_count with money_mention_count - All notebooks parameterized via AWESOMEBITS_DB env var, no real grant data in git-tracked files Refs: awesomefoundation#591
- Add config/signal_score/categories.json with 20 canonical categories in kebab-case - Add AnthropicCache: file-based passthrough cache for API responses, keyed by SHA256 of request params - Add PromptBuilder: assembles system prompt, tool schemas, few-shot examples with chapter-specific config overrides (deep-merge chapter over global default) - Add pre_scorer_spec.rb with unit tests (money extraction, empty fields, TF-IDF) and seeded data tests (tagged :signal_score_data, opt-in via SIGNAL_SCORE_DATA=1) - Prompt supports configurable grant amount and currency for international chapters Refs: awesomefoundation#593, awesomefoundation#591
- categories.json now has slug, name, and description per category - PromptBuilder uses descriptions in tool schema so the LLM knows what each category means - Category tool properties include per-field descriptions Refs: awesomefoundation#593
- Replace hardcoded SCORE_TOOL, SYSTEM_PROMPT, format_application with PromptBuilder calls - Integrate AnthropicCache into batch builder (skip cached requests) - Add --chapter and --no-fewshot flags to build-batch command - build_user_message helper supports both few-shot and rubric-only modes - PromptBuilder is now the single source of truth for prompts Refs: awesomefoundation#593, awesomefoundation#591
- Add signal_score_configs migration: chapter-scoped JSONB config with global default (chapter_id = NULL), partial index - Add SignalScoreConfig model: deep-merge resolution, uniqueness validation, global default constraint - Add SignalScorer: Rails-side entry point wrapping PreScorer and future LLM scoring, reads config from DB - Add signal_score_config_spec with resolve/merge tests Refs: awesomefoundation#593
- SignalScorer service: Anthropic Messages API with Trust Equation - ScoreGrantJob: async SuckerPunch job, fires on project create - Score badge: color-coded score, category, tags, reasoning - Trust Equation breakdown: expandable feature grid - Filter controls: sort by score, min score threshold dropdown - SCSS: uses existing app design tokens Refs: awesomefoundation#590, awesomefoundation#591
- Custom dropdown with jQuery click handler (mirrors chapter-selection) - Gray background, border, CSS triangle arrow - Options as link list with hover states Refs: awesomefoundation#590
All screenshots now show only synthetic Jan 2026 applications. No real grant data visible. Refs: awesomefoundation#590
…rection - Score dropdown now inline with checkbox row - Sort cycles: Score ↕ (off) → Score ↓ (desc) → Score ↑ (asc) - Styled as toggle button matching existing filter controls Refs: awesomefoundation#590
- Score sort toggle and dropdown now use float: left + margin-top: 20px matching the existing short-list-toggle and funded-toggle pattern - Dropdown height matches checkbox height (21px content + 8px padding) - Font size matched to 12px (was 13px) Refs: awesomefoundation#590
- Scores hidden by default, toggled via 'Show scores' checkbox - Sort modes: Default, Score ↓, Score ↑, Latest, Earliest, Random - Random uses stable md5(projects.id) hash — consistent order per page load, no reviewer bias from submission order or score - Sort and filter are now separate dropdowns matching chapter style Refs: awesomefoundation#590
- Remove arrows from sort options - Default sort is 'Newest first' (original created_at DESC) - Rename: 'Score, highest' / 'Score, lowest' / 'Oldest first' Refs: awesomefoundation#590
- Show scores checkbox now toggles via jQuery (no page reload) - Badges always render hidden, JS shows/hides on checkbox change - Sort labels: Latest (default), Earliest, Score highest/lowest, Random - Removed show_scores from URL params (purely client-side state) Refs: awesomefoundation#590
- Fresh screenshots with final UI (scores hidden/shown, sort modes) - README section: Signal Score setup, env vars, how it works, costs - Detail view always shows score badge (full_view flag passthrough) Refs: awesomefoundation#590, awesomefoundation#591
HIGH: - Race condition: atomic claim guard in ScoreGrantJob using WHERE metadata->'signal_score' IS NULL before API call - Lost update: atomic JSONB merge via || operator in persist_score! instead of Ruby hash merge + update_column MEDIUM: - Expression index on composite_score for efficient filtering/sorting - Error body scrubbed from exceptions (only status code + error type) - Retry with exponential backoff (3 attempts, 2s/4s delays) - Failed claims cleaned up so projects can be retried later LOW: - JS var -> const (SonarQube) - Default clause comment on case statement - Sort direction guard via SORT_DIRECTIONS.fetch (prevents injection) Refs: awesomefoundation#590, awesomefoundation#591
- Add webmock gem for HTTP stubbing in tests - Add ProjectFilter specs for signal_score_above, sort_by_signal_score, sort_by_date, sort_by_random (6 new examples) - Fix SignalScorer spec: bypass validation for blank field test - Fix SignalScoreConfig spec: shoulda-matchers v3 compat - Fix ScoreGrantJob retry bug: retry was re-entering the atomic claim check, causing early return instead of actual retry. Extracted score_with_retries! so retries only re-run the scoring call. - Consolidated release_claim! helper to DRY up cleanup paths 34 examples, 0 failures
Contributor
Author
|
A few screenshots Search / List Page with scores visible
Scoring Factor Breakdown (High Score)
Scoring Factor Breakdown (Low Score)
Score Threshold Filter
Sort Order (Thrown in for Good Measure)
|
- Migration: guard metadata column with column_exists? check for CI (squashed 001 migration skips intermediate migrations) - JS: persist 'show scores' checkbox in localStorage - Form: add hidden fields for sort/score_min so form submits preserve dropdown state (shortlist, search, date range)
The implicit 0.15 default filtered out unscored projects, breaking the existing search test and hiding applications before scoring runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an AI-assisted pre-screening system that helps Awesome Foundation trustees prioritize grant applications by surfacing quality signals and red flags. Uses the Trust Equation framework (Credibility + Reliability + Intimacy) / (1 + Self-Interest) to score applications 0.0–1.0.
Not a replacement for human judgment — a triage tool so trustees can focus review time on the most promising applications.
What's Included
Scoring Pipeline
SignalScorer(app/extras/signal_scorer.rb) — Calls Anthropic Messages API (Haiku 4.5, ~$0.02/application) with structuredtool_useoutput for 12 scoring dimensionsScoreGrantJob(app/jobs/score_grant_job.rb) — Async scoring via SuckerPunch with atomic claim (prevents double-scoring), exponential backoff retries, and claim cleanup on failurePreScorer(lib/signal_score/pre_scorer.rb) — Zero-cost deterministic text features (word counts, budget patterns, URL presence)PromptBuilder— Assembles system prompt + tool schema + few-shot examples from DB configConfiguration
SignalScoreConfigmodel — JSONB config with global defaults + chapter-specific deep-merge overridessignal_score_configstable +metadataJSONB column on projectsUI
Filtering & Sorting (
ProjectFilter)signal_score_above(threshold)— JSONB float extraction with index-friendly querysort_by_signal_score— Unscored projects sort last (COALESCE to -1)sort_by_random— Stable random viamd5(id)(no submission-order bias)Data Pipeline (scripts/)
score_grants.rb— Batch scoring with Anthropic cachediscover_patterns.rb— Qualitative pattern analysisimport_data.rb— CSV → DuckDB setup.scratch/cache/anthropic/Tests
34 examples, 0 failures
signal_scorer_spec.rbscore_grant_job_spec.rbsignal_score_config_spec.rbproject_filter_spec.rbAll specs use WebMock — no real API calls in tests.
Bug Fix
Fixed a retry bug in
ScoreGrantJob:retrywas re-entering the atomic claim check at the top of the method, which saw the existing{"status": "scoring"}metadata and returned early — meaning retries never actually happened. Extractedscore_with_retries!to properly scope the retry loop.Validation Results (Chicago chapter, 181 labeled applications)
Not Included
Closes #591, closes #593. Refs #590.