refactor(ledger): split sqlite_store.rs into sub-modules

[fagemx]

Problem

edda-ledger/src/sqlite_store.rs is 7,003 lines with 11 distinct concerns in a single impl SqliteStore block. It is the second largest file and a frequent churn hotspot.

Mixed concerns: connection/schema management, 13 migration functions, device tokens, suggestions, event persistence, references, decision queries, village/pattern detection, bundles, dependencies, causal chain/outcomes, task briefs.

Evidence of Feasibility

device_token.rs was already extracted from this file — the pattern works. The split should continue.

Proposed Structure

src/
├── sqlite_store.rs     (~500 lines: SqliteStore struct, open/create, pragmas)
├── schema/
│   ├── mod.rs          (schema constants)
│   └── migrations.rs   (~600 lines: v1_to_v2 through v11_to_v12)
├── persistence/
│   ├── events.rs       (~460 lines: append, iter, queries)
│   └── snapshots.rs    (~125 lines)
├── entities/
│   ├── suggestions.rs  (~100 lines)
│   ├── bundles.rs      (~115 lines)
│   └── briefs.rs       (~125 lines)
├── decisions/
│   ├── queries.rs      (~220 lines)
│   ├── dependencies.rs (~125 lines)
│   └── outcomes.rs     (~320 lines)
└── analytics/
    └── village.rs      (~315 lines: stats, pattern detection)

Notes

• Pure refactor — no behavior change
• Tests (~3,400 lines) should be split to match
• device_token.rs extraction is the template to follow

[fagemx]

Research: Split sqlite_store.rs into Sub-modules (GH-376)

Current State

edda-ledger/src/sqlite_store.rs is 7,003 lines — the second largest file in the workspace. It contains a single impl SqliteStore block with 11 distinct concerns, plus ~3,400 lines of tests.

File Anatomy (Line Ranges)

Concern	Lines	Range	Description
Schema constants	~210	1–211	12 SCHEMA_* SQL constants (V1–V12)
Type definitions	~258	213–470	15 public structs/enums + ImportParams + status_to_is_active()
Core struct + open	~30	471–504	SqliteStore { conn }, open(), open_or_create(), apply_pragmas()
Schema management	~100	506–705	apply_schema(), schema_version(), set_schema_version(), verify_decisions_schema()
Migrations	~395	706–1098	12 migration functions (migrate_v1_to_v2 through migrate_v11_to_v12) + backfill_task_brief_updates()
Device tokens	~100	1101–1201	5 CRUD functions
Suggestions	~100	1203–1301	4 CRUD functions
Events	~540	1303–1840	append_event, append_event_idempotent, iter_events*, find_related_*, get_event, events_after_rowid, refs operations
Decisions	~730	1841–2543	active_decisions, decision_timeline, domain_timeline, list_domains, village_stats, find_active_decision, cross-project sync
Pattern detection	~160	2206–2367	detect_village_patterns
Bundles	~45	2474–2518	get_bundle, list_bundles
Dependencies	~125	2544–2667	insert_dep, deps_of, dependents_of, transitive_dependents_of, active_dependents_of
Causal chain	~170	2668–2833	get_decision_by_event_id, causal_chain
Outcomes	~165	2835–3000	decision_outcomes, executions_for_decision
Chain verification	~30	3002–3039	verify_chain
Task briefs	~60	3041–3099	get_task_brief, list_task_briefs
Snapshots	~85	3101–3185	insert_snapshot, query_snapshots, snapshots_by_context_hash
Free functions	~445	3187–3627	Row mappers, materialization helpers, Drop impl, EventRow, task brief update helpers
Tests	~3,374	3629–7003	70+ test functions

Public API Surface

Types re-exported from lib.rs:

pub use sqlite_store::{
    BundleRow, ChainEntry, DecideSnapshotRow, DecisionRow, DepRow,
    DetectedPattern, DeviceTokenRow, ImportParams, PatternDetectionResult,
    PatternType, SuggestionRow, TaskBriefRow,
};

Types NOT re-exported but used internally:

• VillageStats, VillageStatsPeriod, DomainCount, DayCount
• OutcomeMetrics, ExecutionLinked
• EventRow (private)

Internal Dependencies

The SqliteStore struct holds only conn: Connection. All methods operate through self.conn.

Cross-concern calls within impl SqliteStore:

• append_event() calls materialize_bundle_sql(), materialize_task_brief_sql(), update_task_brief_on_commit(), update_task_brief_on_note(), update_task_brief_on_merge() (all free functions taking &Connection)
• append_event() calls edda_core::decision::* for decision materialization
• decision_outcomes() calls self.get_event() and self.executions_for_decision()
• causal_chain() calls self.get_decision_by_event_id()
• transitive_dependents_of() calls self.active_dependents_of()
• backfill_task_brief_updates() uses extract_artifacts_from_payload(), payload_tags(), extract_feedback_from_payload()
• village_stats() and detect_village_patterns() are independent query methods
• detect_village_patterns() calls detect_trend_direction() (free function)

External Callers

The main external caller is Ledger in ledger.rs, which wraps SqliteStore as a thin delegation layer with .context() error decoration. Every public SqliteStore method is called through Ledger.

Other external callers:

• edda-ask — imports DecisionRow directly from sqlite_store
• edda-serve — uses VillageStats, PatternDetectionResult, DeviceTokenRow via full path
• sync.rs (within crate) — uses ImportParams
• view.rs (within crate) — uses DecisionRow

Precedent: device_token.rs Extraction

The device_token.rs module (65 lines) was previously extracted from sqlite_store.rs. It contains standalone utility functions (generate_device_token, hash_token) with no dependency on SqliteStore — it's a pure function extraction, not a sub-module of impl SqliteStore.

This is distinct from what issue #376 proposes, which is splitting impl SqliteStore methods into sub-modules.

Recent Changes (PR #381)

PR #381 added .context() calls throughout ledger.rs, meaning every Ledger method now wraps its SqliteStore call with .context("Ledger::method_name"). This doesn't affect the split plan but means ledger.rs has recently been modified.

Related Issues

• [refactor] Split god file sqlite_store.rs (6,099 lines) #352: Older version of same issue (less detail, created by tech-debt scan)
• refactor(ledger): wrap sqlite_store types in domain abstractions #378: Proposes wrapping sqlite_store types in domain abstractions — complementary but separate concern
• refactor(serve): split edda-serve/src/lib.rs into sub-modules #375: Similar split for edda-serve/src/lib.rs

[fagemx]

Innovation: Split sqlite_store.rs into Sub-modules (GH-376)

Key Tension

The central challenge: how to split impl SqliteStore methods across files while keeping a single struct?

Rust supports splitting impl blocks across files within the same crate, but only within the same module or its sub-modules. The methods need self.conn access, and some methods call other methods on self.

Approach A: Sub-modules with impl SqliteStore in each file

Mechanism: Create a sqlite_store/ directory with mod.rs containing the struct definition and open/open_or_create. Each sub-module file adds more impl SqliteStore blocks.

sqlite_store/
├── mod.rs          (struct, open, pragmas, Drop, re-exports)
├── schema.rs       (schema constants, apply_schema, migrations)
├── events.rs       (append, iter, get, find_related)
├── decisions.rs    (active_decisions, timeline, domain_timeline, find_active)
├── village.rs      (village_stats, detect_village_patterns)
├── dependencies.rs (insert_dep, deps_of, dependents, transitive)
├── causal.rs       (get_decision_by_event_id, causal_chain)
├── outcomes.rs     (decision_outcomes, executions_for_decision)
├── bundles.rs      (get_bundle, list_bundles)
├── briefs.rs       (get_task_brief, list_task_briefs, materialization helpers)
├── snapshots.rs    (insert_snapshot, query_snapshots)
├── suggestions.rs  (insert_suggestion, list, get, update)
├── device_tokens.rs(insert, validate, list, revoke)
├── types.rs        (all row structs, enums)
└── mappers.rs      (map_*_row, row_to_event, EventRow, time helpers)

Pros:

• Each file is a cohesive unit around one concern
• No API change — sqlite_store::SqliteStore path unchanged
• mod.rs re-exports everything, external code unaffected
• Cross-method calls (self.get_event() from outcomes.rs) work naturally since they're all impl SqliteStore

Cons:

• Sub-module files need use super::* or explicit imports to access SqliteStore, Connection, etc.
• IDE support for "go to definition" may be slightly less convenient with scattered impl blocks

Verdict: This is the idiomatic Rust pattern. The standard library does this (e.g., str methods split across files). This is the strongest approach.

Approach B: Trait-based extraction

Mechanism: Define traits per concern (e.g., trait DecisionQueries, trait EventPersistence) and implement them for SqliteStore.

// decisions.rs
pub trait DecisionQueries {
    fn active_decisions(&self, ...) -> ...;
}
impl DecisionQueries for SqliteStore { ... }

Pros:

• Clean interfaces per concern
• Could enable future backend swapping

Cons:

• Over-engineering for a pure refactor — adds abstraction where none is needed
• Callers would need to use DecisionQueries trait import
• Breaks existing API (store.active_decisions() requires trait in scope)
• Violates project's YAGNI principle
• ledger.rs and all callers would need updating

Verdict: Rejected. This adds complexity without solving the actual problem (file size). It's a different concern (addressed by #378).

Approach C: Flat files (no sub-directory)

Mechanism: Keep sqlite_store.rs as-is but add sibling files like sqlite_decisions.rs, sqlite_events.rs.

Pros:

• Simpler directory structure

Cons:

• Can't split impl SqliteStore across different modules without the sub-module relationship
• Would require making SqliteStore.conn pub(crate) and using free functions
• Loses method call syntax (self.get_event() becomes get_event(&self.conn))
• Breaks all existing callers

Verdict: Rejected. Doesn't work well with Rust's module system for impl blocks.

Recommended: Approach A with Pragmatic Groupings

Refined Module Layout

Based on the research, some concerns are too small to justify standalone files. Pragmatic groupings:

sqlite_store/
├── mod.rs           (~200 lines: struct, open/create, pragmas, Drop, re-exports)
├── types.rs         (~280 lines: all row structs, enums, ImportParams)
├── schema.rs        (~600 lines: constants + apply_schema + 12 migrations + verify)
├── events.rs        (~540 lines: append, iter, get, find_related, refs, chain verify)
├── decisions.rs     (~600 lines: active_decisions, timeline, domain, find_active, sync)
├── village.rs       (~300 lines: village_stats + detect_village_patterns + detect_trend_direction)
├── dependencies.rs  (~250 lines: insert_dep, deps/dependents, causal_chain, outcomes)
├── entities.rs      (~350 lines: bundles, briefs, snapshots, suggestions, device_tokens)
└── mappers.rs       (~250 lines: map_*_row, row_to_event, EventRow, materialization helpers)

8 files instead of the proposed 13 — pragmatically grouped to avoid files under 100 lines.

Key Design Decisions

1. dependencies.rs includes causal chain and outcomes: These all relate to decision relationships and share the pattern of traversing links between decisions.

2. entities.rs groups small CRUD concerns: Bundles (45 lines), briefs (60 lines), snapshots (85 lines), suggestions (100 lines), and device tokens (100 lines) are all simple CRUD that would be tiny standalone files. Grouping them keeps each section clearly delimited with // ── Section ── markers.

3. mappers.rs centralizes all row mapping: All map_*_row functions, row_to_event, EventRow struct, materialize_*_sql helpers, and task brief update helpers. These are consumed by multiple other modules.

4. types.rs is separate from mod.rs: Keeps the struct definitions browseable without scrolling past connection logic.

5. Tests: Each module gets its own #[cfg(test)] mod tests block with the relevant test functions moved alongside the code they test. The shared tmp_db() helper and COUNTER go into a test_helpers sub-module or mod.rs tests.

Cross-Module Dependencies

mod.rs ← (defines SqliteStore, pub(crate) conn)
  ↓
types.rs (standalone, no SqliteStore dependency)
schema.rs → types (migration uses type defs)
mappers.rs → types (row mappers return types)
events.rs → mappers (uses map_event_row, row_to_event)
decisions.rs → mappers (uses map_decision_row), types
village.rs → types (VillageStats, etc.)
dependencies.rs → mappers, types, decisions (calls get_decision_by_event_id)
entities.rs → mappers, types

The conn field needs to be pub(crate) so sub-modules can access it via self.conn.

Migration Safety

• No behavior change: Pure code move, identical SQL, identical types
• No API change: sqlite_store::SqliteStore path unchanged, all re-exports preserved
• Verification: cargo test -p edda-ledger must pass identically before and after

[fagemx]

Plan: Split sqlite_store.rs into Sub-modules (GH-376)

Strategy

Convert sqlite_store.rs (7,003 lines) from a single file into a sqlite_store/ directory module using Rust's split-impl-block pattern. Each sub-module adds its own impl SqliteStore block. No behavior or API change.

Target Layout

crates/edda-ledger/src/
├── sqlite_store/
│   ├── mod.rs           (~200 lines)
│   ├── types.rs         (~280 lines)
│   ├── schema.rs        (~600 lines)
│   ├── events.rs        (~540 lines)
│   ├── decisions.rs     (~600 lines)
│   ├── village.rs       (~300 lines)
│   ├── dependencies.rs  (~250 lines)
│   ├── entities.rs      (~350 lines)
│   └── mappers.rs       (~250 lines)
└── (existing files unchanged)

File Contents

mod.rs (~200 lines)

• Module declarations and re-exports
• SqliteStore struct with pub(crate) conn: Connection
• open(), open_or_create(), apply_pragmas()
• Drop impl (WAL checkpoint)
• status_to_is_active() free function

types.rs (~280 lines)

• All public structs: DecisionRow, ChainEntry, BundleRow, DepRow, VillageStats, VillageStatsPeriod, PatternType, DetectedPattern, PatternDetectionResult, DomainCount, DayCount, ImportParams, TaskBriefRow, DeviceTokenRow, DecideSnapshotRow, SuggestionRow, OutcomeMetrics, ExecutionLinked

schema.rs (~600 lines)

• All SCHEMA_*_SQL constants (V1-V12)
• apply_schema(), schema_version(), set_schema_version(), verify_decisions_schema()
• All 12 migrate_v*_to_v*() functions
• backfill_task_brief_updates()

events.rs (~540 lines)

• append_event(), append_event_idempotent()
• iter_events(), iter_events_by_type(), iter_branch_events(), iter_events_filtered()
• find_related_commits(), find_related_notes()
• get_event(), events_after_rowid(), rowid_for_event_id()
• last_event_hash()
• head_branch(), set_head_branch(), branches_json(), set_branches_json()
• verify_chain()

decisions.rs (~600 lines)

• active_decisions(), active_decisions_with_paths()
• decision_timeline(), domain_timeline(), list_domains()
• find_active_decision(), get_decision_by_event_id()
• shared_decisions(), is_already_imported(), insert_imported_decision()

village.rs (~300 lines)

• village_stats()
• detect_village_patterns()
• detect_trend_direction() free function

dependencies.rs (~250 lines)

• insert_dep(), deps_of(), dependents_of()
• transitive_dependents_of(), active_dependents_of()
• causal_chain()
• decision_outcomes(), executions_for_decision()

entities.rs (~350 lines)

• Device tokens: insert_device_token(), validate_device_token(), list_device_tokens(), revoke_device_token(), revoke_all_device_tokens()
• Suggestions: insert_suggestion(), list_suggestions_by_status(), get_suggestion(), update_suggestion_status()
• Bundles: get_bundle(), list_bundles()
• Task briefs: get_task_brief(), list_task_briefs()
• Snapshots: insert_snapshot(), query_snapshots(), snapshots_by_context_hash()

mappers.rs (~250 lines)

• EventRow struct (private)
• map_event_row(), row_to_event()
• map_decision_row(), map_bundle_row(), map_dep_row(), map_task_brief_row(), map_snapshot_row()
• materialize_bundle_sql(), materialize_task_brief_sql()
• update_task_brief_on_commit(), update_task_brief_on_note(), update_task_brief_on_merge()
• extract_artifacts_from_payload(), payload_tags(), extract_feedback_from_payload()
• time_now_rfc3339()

Implementation Steps

Step 1: Baseline (pre-check)

cargo fmt --check -p edda-ledger
cargo clippy -p edda-ledger -- -D warnings
cargo test -p edda-ledger

Record test count and pass rate.

Step 2: Create directory structure

mkdir crates/edda-ledger/src/sqlite_store

Step 3: Create types.rs

Move all struct/enum definitions (lines 213-470) to sqlite_store/types.rs.

• Add necessary imports: use serde, use edda_core::types::*
• Keep all #[derive] attributes intact

Step 4: Create mappers.rs

Move all row mapping functions and helpers (lines 3187-3627) to sqlite_store/mappers.rs.

• Add use super::types::*
• Add use rusqlite::{params, Connection}
• Add use edda_core::types::*
• Make EventRow pub(super) so events.rs can use it
• Make mapper functions pub(super) for use by other sub-modules

Step 5: Create schema.rs

Move schema constants (lines 12-211) and migration functions (lines 506-1098) to sqlite_store/schema.rs.

• Add use super::SqliteStore
• Add impl SqliteStore { ... } block with all migration methods
• Move backfill_task_brief_updates here too (it's part of v6 migration)
• Needs access to mappers::extract_artifacts_from_payload, mappers::payload_tags, mappers::extract_feedback_from_payload

Step 6: Create events.rs

Move event persistence methods (lines 1303-1840, 3002-3039) to sqlite_store/events.rs.

• Add use super::{SqliteStore, mappers::*}
• append_event needs access to materialize_bundle_sql, materialize_task_brief_sql, task brief update helpers (all in mappers)
• iter_events* need map_event_row, row_to_event

Step 7: Create decisions.rs

Move decision query methods (lines 1841-2475) to sqlite_store/decisions.rs.

• Includes active_decisions, timeline queries, cross-project sync methods
• Uses map_decision_row from mappers

Step 8: Create village.rs

Move village stats and pattern detection (lines 2077-2367) to sqlite_store/village.rs.

• Move detect_trend_direction free function here
• Uses types from types.rs

Step 9: Create dependencies.rs

Move dependency, causal chain, and outcome methods (lines 2544-3000) to sqlite_store/dependencies.rs.

• Uses map_dep_row, map_decision_row from mappers
• causal_chain calls self.get_decision_by_event_id() -- works because it's impl SqliteStore in a different file
• decision_outcomes calls self.get_event() and self.executions_for_decision() -- same pattern

Step 10: Create entities.rs

Move CRUD methods for device tokens (1101-1201), suggestions (1203-1301), bundles (2474-2518), briefs (3041-3099), snapshots (3101-3185) to sqlite_store/entities.rs.

• Uses various mapper functions
• Each section delimited with // ── Section ── markers

Step 11: Create mod.rs

Create sqlite_store/mod.rs with:

• mod declarations for all sub-modules
• pub use types::* to maintain API compatibility
• SqliteStore struct with pub(crate) conn
• open(), open_or_create(), apply_pragmas()
• Drop impl
• status_to_is_active()

Step 12: Delete old sqlite_store.rs

Remove the original file (replaced by sqlite_store/mod.rs).

Step 13: Update lib.rs

The pub mod sqlite_store; declaration should work unchanged (Rust auto-discovers sqlite_store/mod.rs). Verify re-exports still compile.

Step 14: Split tests

Move tests from the old monolithic mod tests into per-module test blocks:

• Shared helpers (tmp_db, COUNTER, make_decision_event) go into mod.rs tests or a shared test helper
• Schema tests → schema.rs
• Event tests → events.rs
• Decision tests → decisions.rs
• Village/pattern tests → village.rs
• Dependency/causal/outcome tests → dependencies.rs
• Entity tests (bundles, briefs, snapshots, device tokens, suggestions) → entities.rs

Test helper visibility: tmp_db() and helper event factories are needed across multiple test modules. Options:

• Option A: Create a #[cfg(test)] pub(crate) mod test_helpers in mod.rs
• Option B: Duplicate tmp_db() in each test module (simple but repetitive)
• Recommended: Option A — centralize in mod.rs

Step 15: Post-check (regression)

cargo fmt --check -p edda-ledger
cargo clippy -p edda-ledger -- -D warnings
cargo test -p edda-ledger
cargo test --workspace  # full workspace to catch cross-crate breakage

Verify same test count and all pass.

Execution Order Rationale

The steps are ordered by dependency:

1. types.rs first (no dependencies)
2. mappers.rs second (depends on types only)
3. schema.rs third (depends on types, mappers)
4. Everything else in any order (all depend on types + mappers)
5. mod.rs last (ties everything together)
6. Tests last (need all code in place)

Risk Mitigation

Risk	Mitigation
Visibility errors	conn becomes pub(crate); mapper functions become pub(super)
Cross-method calls break	All impl SqliteStore blocks work across files within same module
Import path changes break external callers	mod.rs re-exports everything at same paths
Tests can't access private helpers	#[cfg(test)] pub(crate) test helper module
Merge conflicts with in-flight PRs	Pure refactor on separate branch; merge main first

Verification Criteria

• cargo test -p edda-ledger passes with identical test count
• cargo test --workspace passes
• cargo clippy --workspace -- -D warnings passes
• No public API changes (same re-exports in lib.rs)
• Each sub-module file is under 600 lines
• sqlite_store.rs no longer exists (replaced by directory)

Scope Boundary

In scope: Moving code between files within edda-ledger/src/sqlite_store/.
Out of scope:

• refactor(ledger): wrap sqlite_store types in domain abstractions #378 (domain abstraction / type wrapping) — separate concern, do later
• [refactor] Split god file sqlite_store.rs (6,099 lines) #352 (close as duplicate of refactor(ledger): split sqlite_store.rs into sub-modules #376)
• Changing any SQL queries, schemas, or behavior
• Modifying ledger.rs or external callers

[fagemx]

Work completed. PR created: #382

Split sqlite_store.rs (7,003 lines) into 9 sub-module files under sqlite_store/ directory. Pure refactor — no behavior or API changes. All 159 tests pass identically.