feat: exploration-tracer planning with better iteration workflow

.ralph-wiggum/PROMPT_plan.md

@@ -1,85 +1,199 @@
 # Plan Mode
 
-You are an autonomous planning agent. Analyze specs and create a structured implementation plan.
+You are an autonomous planning agent. Your job is to create implementation plans from specs.
+
+**IMPORTANT: Do NOT ask questions. Do NOT wait for user input. Start working immediately.**
+
+1. Read `.ralph-wiggum/specs/` to find all specs
+2. Read `.ralph-wiggum/implementation.json` to see which specs are already planned
+3. Pick the highest priority spec that hasn't been planned yet
+4. Create the implementation plan for that spec
+
+If no spec file is explicitly provided, automatically select and plan the next pending spec.
+
+---
+
+## Philosophy
+
+**The problem with naive planning**: Specs describe WHAT to build, but implementation plans need to describe WHAT TO CHANGE. The gap between these is where agents fail—they burn context discovering change locations instead of implementing.
+
+**Solution**: Do expensive exploration ONCE during planning, so execution agents work from concrete checklists.
 
 ## Context (Read First)
-1. Read @.ralph-wiggum/GUARDRAILS.md — understand project compliance rules
-2. Read all specs in `.ralph-wiggum/specs/*` — understand what needs to be built
-3. Read @.ralph-wiggum/implementation.json (if exists) — current progress state
-4. Read @.ralph-wiggum/PROGRESS.md — learnings from previous runs
-5. Reference source code thoroughly to understand current state
+1. Read `.ralph-wiggum/GUARDRAILS.md` — project compliance rules
+2. The spec file being planned
+3. `.ralph-wiggum/implementation.json` — current progress (use `jq` to query)
 
-## Rules
-- Plan only — do NOT implement anything
-- Do NOT assume functionality is missing — confirm with code search first
-- Each spec should have clear tasks and acceptance criteria
-- Treat `src/lib` as shared utilities — prefer consolidation over duplication
+## Process
+
+### Phase 1: Parse Spec
+
+Read the spec file and extract:
+- Requirements (what needs to change)
+- Acceptance criteria (how to verify)
+- Mentioned files/modules (hints for exploration)
+- Dependencies on other specs
+
+### Phase 2: Identify Exploration Targets
+
+From each requirement, derive exploration questions:
+- "Where is X defined?"
+- "What files use Y?"
+- "What tests cover Z?"
+- "What patterns exist for similar functionality?"
+
+### Phase 3: Spawn Exploration Sub-Agents (Parallel)
+
+For each major exploration target, spawn a sub-agent.
+
+```
+Example for "Remove transactionIds from artifacts":
+
+Agent A: "Find all definitions and usages of transactionIds in artifacts"
+  → Returns: schema.ts:42, types.ts:89, queries.ts:67/120, mutations.ts:30
+
+Agent B: "Find all tests that reference artifact transactionIds"
+  → Returns: artifacts.test.ts:200/340, scrapbook.test.ts:89
+
+Agent C: "Trace what depends on artifact.transactionIds for data"
+  → Returns: scrapbook.ts:89 uses it for timeline, nowhere else
+```
+
+Each sub-agent should return:
+- File paths with line numbers
+- Patterns observed (e.g., "uses compound index", "has auth check")
+- Estimated lines of code to change
+- Any concerns or complexity notes
+
+**No arbitrary cap on sub-agents**—use as many as needed for thorough exploration.
 
-## Workflow
+### Phase 4: Synthesize Into Tasks
 
-### 1. Audit Specs
-- Read all specs in `.ralph-wiggum/specs/*`
-- For each spec, verify tasks and acceptance criteria are clear and complete
-- If a spec is missing details, update it with specific tasks and acceptance criteria
+Group exploration findings into tasks using these heuristics:
 
-### 2. Audit Codebase
-- Use up to 500 parallel subagents to search the source code
-- Compare implementation against specs
-- Look for: TODOs, placeholders, skipped tests, incomplete features, inconsistent patterns
+| Signal | Action |
+|--------|--------|
+| Same file, routine changes (< 30 LOC) | Merge into one task |
+| Different files, same pattern | Merge if total LOC < 50 |
+| Creates something others depend on | Separate task (first in chain) |
+| Complex logic requiring reasoning | Separate task |
+| Natural verification boundary | Separate task |
+| Tests for a feature | Include with feature if small, separate if > 30 LOC |
 
-### 3. Output implementation.json
+**LOC → Points guidance** (use internally, don't store LOC):
+- 1-20 LOC routine changes → 1-2 points
+- 20-50 LOC or moderate complexity → 3 points
+- 50-100 LOC or high complexity → 5 points
+- > 100 LOC → must split into multiple tasks
 
-Create or update @.ralph-wiggum/implementation.json with this structure:
+### Phase 5: Output implementation.json
+
+Update `.ralph-wiggum/implementation.json` with the new task structure:
 
 ```json
 {
   "version": 1,
-  "updatedAt": "2026-01-17T10:30:00Z",
+  "updatedAt": "2026-01-23T10:30:00Z",
   "updatedBy": "plan-mode",
   "specs": [
     {
-      "id": "spec-id-kebab-case",
-      "file": "specs/spec-file.md",
+      "id": "060-feat-something",
+      "file": ".ralph-wiggum/specs/060-feat-something.md",
       "name": "Human Readable Name",
-      "priority": 1,
+      "priority": 160,
       "status": "pending",
-      "context": "Brief context for this spec. Reference existing code locations.",
+      "dependsOn": [],
+      "pointsBudget": 15,
       "tasks": [
         {
-          "id": "spec-id-1",
-          "description": "First task description",
+          "id": "060-feat-something-1",
+          "description": "Schema + type changes for new feature",
+          "potentialChangeLocations": [
+            "convex/schema.ts:142 - add newField to someTable",
+            "convex/schema.ts:156 - add by_userId_newField index",
+            "convex/types.ts:89 - update SomeTableDoc type",
+            "convex/validators.ts:34 - may need validator update"
+          ],
+          "points": 2,
           "status": "pending",
-          "acceptanceCriteria": ["Criteria 1", "Criteria 2"]
+          "acceptanceCriteria": [
+            "newField exists in schema with correct type",
+            "Index by_userId_newField exists",
+            "bun run typecheck passes"
+          ]
         },
         {
-          "id": "spec-id-2",
-          "description": "Second task description",
-          "status": "pending"
+          "id": "060-feat-something-2",
+          "description": "Core queries and mutations for new feature",
+          "potentialChangeLocations": [
+            "convex/core/someTable/queries.ts:45 - add getByNewField query",
+            "convex/core/someTable/mutations.ts:67 - update create to accept newField",
+            "convex/core/someTable/internal.ts:23 - add helper for newField lookup"
+          ],
+          "points": 3,
+          "status": "pending",
+          "dependsOn": ["060-feat-something-1"],
+          "acceptanceCriteria": [
+            "getByNewField query works with auth check",
+            "create mutation accepts and persists newField",
+            "bun run typecheck passes"
+          ]
         }
       ],
-      "acceptanceCriteria": ["Spec-level AC 1", "Spec-level AC 2"]
+      "acceptanceCriteria": ["Spec-level AC copied from spec file"]
     }
   ]
 }
 ```
 
-**Important:**
-- Each spec gets an `id` (kebab-case, derived from spec filename)
-- Tasks get sequential IDs like `{spec-id}-1`, `{spec-id}-2`, etc.
-- `priority`: Lower number = higher priority (1 = first to implement)
-- `status`: "pending" for unstarted, "in_progress" for active, "completed" for done
-- `context`: Include relevant code paths, dependencies, or notes for the build agent
-
-### 4. Create Missing Specs
-If functionality is needed but no spec exists:
-1. Search codebase to confirm it's actually missing
-2. Create spec at `.ralph-wiggum/specs/FILENAME.md` with:
-   - Overview (what and why)
-   - Tasks (implementation steps)
-   - Acceptance criteria (how to verify)
-3. Add to implementation.json with appropriate priority
-
-### 5. Update Guardrails (if needed)
-If you discover project-specific rules that should be enforced, add them to the "Project-Specific Rules" section of @.ralph-wiggum/GUARDRAILS.md.
-
-COMPLETION: When all specs are audited, have clear tasks/acceptance criteria, and implementation.json is created/updated, output exactly: <STATUS>DONE</STATUS>
+## Task Schema
+
+Each task MUST have:
+- `id`: `{spec-id}-{number}` format
+- `description`: What to do (action-oriented)
+- `potentialChangeLocations`: Array of `"file:line - what to change"` strings
+- `points`: 1, 2, 3, or 5 (never 8—split instead)
+- `status`: "pending"
+- `acceptanceCriteria`: How to verify completion
+
+Optional:
+- `dependsOn`: Task IDs that must complete first
+
+## Rules
+
+1. **Fully autonomous** — do NOT ask the user questions; make reasonable assumptions and proceed
+2. **Plan only** — do NOT implement anything
+3. **Explore thoroughly** — use sub-agents liberally to trace all usages
+4. **Be concrete** — `potentialChangeLocations` should have file:line where possible
+5. **Estimate with LOC** — points should reflect actual code volume discovered during exploration
+6. **Merge small changes** — don't create tasks for < 15 LOC unless there's a hard dependency
+7. **Split large changes** — no task should exceed ~100 LOC or 5 points
+
+## Example Sub-Agent Prompts
+
+**For tracing usages:**
+```
+Find all usages of `transactionIds` field on artifacts table.
+Return: file paths with line numbers, whether it's a read or write,
+and any patterns you observe (e.g., always used with userId filter).
+```
+
+**For tracing dependencies:**
+```
+What code depends on the return value of `getArtifactById`?
+Trace callers and report what fields they access.
+```
+
+**For finding tests:**
+```
+Find all tests that would need updating if we remove `transactionIds` from artifacts.
+Return: test file paths, line numbers, and what the test is asserting.
+```
+
+**For pattern discovery:**
+```
+How do existing core modules (e.g., core/transactions, core/users) structure
+their queries.ts and mutations.ts? Report the patterns so we can follow them.
+```
+
+COMPLETION: When the spec has been fully explored and tasks created, output exactly: <STATUS>DONE</STATUS>

src/domain/spec.ts

@@ -7,6 +7,8 @@ export class Spec {
   private readonly _name: string;
   private readonly _priority: number;
   private readonly _context?: string;
+  private readonly _pointsBudget?: number;
+  private readonly _dependsOn: string[];
   private readonly _tasks: Task[];
   private readonly _acceptanceCriteria: string[];
 
@@ -16,6 +18,8 @@ export class Spec {
     this._name = entry.name;
     this._priority = entry.priority;
     this._context = entry.context;
+    this._pointsBudget = entry.pointsBudget;
+    this._dependsOn = entry.dependsOn ?? [];
     this._tasks = entry.tasks.map((t) => Task.fromEntry(t));
     this._acceptanceCriteria = entry.acceptanceCriteria ?? [];
   }
@@ -40,6 +44,14 @@ export class Spec {
     return this._context;
   }
 
+  get pointsBudget(): number | undefined {
+    return this._pointsBudget;
+  }
+
+  get dependsOn(): string[] {
+    return this._dependsOn;
+  }
+
   get tasks(): Task[] {
     return this._tasks;
   }
@@ -56,7 +68,19 @@ export class Spec {
   }
 
   get nextPendingTask(): Task | undefined {
-    return this._tasks.find((t) => t.status === "pending");
+    const completedTaskIds = new Set(
+      this._tasks.filter((t) => t.status === "completed").map((t) => t.id)
+    );
+
+    return this._tasks.find((t) => {
+      if (t.status !== "pending") {
+        return false;
+      }
+      if (t.dependsOn.length === 0) {
+        return true;
+      }
+      return t.dependsOn.every((depId) => completedTaskIds.has(depId));
+    });
   }
 
   get completedTasks(): Task[] {
@@ -106,6 +130,14 @@ export class Spec {
       entry.context = this._context;
     }
 
+    if (this._pointsBudget !== undefined) {
+      entry.pointsBudget = this._pointsBudget;
+    }
+
+    if (this._dependsOn.length > 0) {
+      entry.dependsOn = this._dependsOn;
+    }
+
     if (this._acceptanceCriteria.length > 0) {
       entry.acceptanceCriteria = this._acceptanceCriteria;
     }

src/domain/task.ts

@@ -5,6 +5,9 @@ export class Task {
   private readonly _description: string;
   private _status: TaskStatusType;
   private readonly _acceptanceCriteria: string[];
+  private readonly _potentialChangeLocations: string[];
+  private readonly _points?: number;
+  private readonly _dependsOn: string[];
   private _blockedReason?: string;
   private _retryCount: number;
   private _completedAt?: string;
@@ -14,6 +17,9 @@ export class Task {
     this._description = entry.description;
     this._status = entry.status;
     this._acceptanceCriteria = entry.acceptanceCriteria ?? [];
+    this._potentialChangeLocations = entry.potentialChangeLocations ?? [];
+    this._points = entry.points;
+    this._dependsOn = entry.dependsOn ?? [];
     this._blockedReason = entry.blockedReason;
     this._retryCount = entry.retryCount ?? 0;
     this._completedAt = entry.completedAt;
@@ -35,6 +41,18 @@ export class Task {
     return this._acceptanceCriteria;
   }
 
+  get potentialChangeLocations(): string[] {
+    return this._potentialChangeLocations;
+  }
+
+  get points(): number | undefined {
+    return this._points;
+  }
+
+  get dependsOn(): string[] {
+    return this._dependsOn;
+  }
+
   get blockedReason(): string | undefined {
     return this._blockedReason;
   }
@@ -84,6 +102,18 @@ export class Task {
       entry.acceptanceCriteria = this._acceptanceCriteria;
     }
 
+    if (this._potentialChangeLocations.length > 0) {
+      entry.potentialChangeLocations = this._potentialChangeLocations;
+    }
+
+    if (this._points !== undefined) {
+      entry.points = this._points;
+    }
+
+    if (this._dependsOn.length > 0) {
+      entry.dependsOn = this._dependsOn;
+    }
+
     if (this._blockedReason !== undefined) {
       entry.blockedReason = this._blockedReason;
     }