feat: integrate Drain3-style log template mining into audit report and logs

[Copilot]

Summary

Implements a production-oriented Drain3-style log template mining module and integrates it into the gh aw audit, gh aw audit report, and gh aw logs observability pipeline. Includes a --train flag on the logs command to build and embed default weights.

What this PR does

New package: pkg/agentdrain

A standalone, zero-dependency Go package implementing online Drain3-style log template mining, specifically designed for agent session observability:

File	Purpose
types.go	Exported types: Config, MaskRule, Cluster, MatchResult, AnomalyReport, AgentEvent
config.go	DefaultConfig() with sensible defaults and built-in masking rules
mask.go	Masker (UUID, session ID, number, URL, timestamp, quoted-string), FlattenEvent, Tokenize
tree.go	Fixed-depth Drain parse tree (token-count bucket → first-token → cluster IDs)
cluster.go	clusterStore, computeSimilarity, mergeTemplate
miner.go	Thread-safe Miner: Train / Match / TrainEvent / AnalyzeEvent
anomaly.go	AnomalyDetector — weighted anomaly score (new=1.0, low-sim=0.7, rare=0.3)
pretrain.go	PreTrainTemplate / PreTrainTemplates / PreTrainTemplateCounts
persist.go	JSON round-trip: SaveJSON / LoadJSON / LoadMinerJSON
coordinator.go	Stage-based Coordinator (plan, tool_call, tool_result, retry, error, finish) + StageSequence + SaveWeightsJSON / LoadWeightsJSON
defaults.go	Embeds data/default_weights.json; LoadDefaultWeights() preloads embedded templates
miner_test.go	12 tests: cluster creation/merge, inference-only, masking, flatten, pretrain, save/load, concurrency, stage routing, similarity, merge
anomaly_test.go	5 tests: new template, low similarity, rare cluster, normal match, full pipeline

New: pkg/cli/drain3_integration.go

Integration layer connecting pkg/agentdrain to the existing audit pipeline. Coordinators call LoadDefaultWeights() on startup so trained templates are available immediately:

• buildDrain3Insights(processedRun, metrics, toolUsage) — converts a single run's structured data into AgentEvents, mines templates, and returns ObservabilityInsights covering cluster summary, anomaly report, and stage sequence evidence.
• buildDrain3InsightsMultiRun(processedRuns) — same for cross-run analysis with a shared coordinator, detecting patterns across ≤50 runs.
• buildDrain3InsightsFromCrossRunInputs(inputs) — converts crossRunInput entries to ProcessedRuns and delegates to buildDrain3InsightsMultiRun, used by the audit report subcommand.

New: pkg/cli/drain3_train.go

TrainDrain3Weights(processedRuns, outputDir, verbose) trains a Drain3 coordinator across all processed runs and writes drain3_weights.json to the output directory. The file can be committed to pkg/agentdrain/data/default_weights.json and rebuilt to embed it as the binary default.

Modified: pkg/cli/logs_command.go — --train flag

gh aw logs --train                    # train on last 10 runs
gh aw logs my-workflow --train -c 50  # train on up to 50 runs

When --train is set, DownloadWorkflowLogs runs TrainDrain3Weights after processing all runs and returns an error if training fails.

Modified: pkg/cli/audit_report.go

Appends Drain3 insights to ObservabilityInsights in buildAuditData():

observabilityInsights := buildAuditObservabilityInsights(...)
observabilityInsights = append(observabilityInsights, buildDrain3Insights(...)...)

Modified: pkg/cli/logs_report.go

Appends Drain3 multi-run insights to Observability in BuildLogsData():

observability := buildLogsObservabilityInsights(processedRuns, toolUsage)
observability = append(observability, buildDrain3InsightsMultiRun(processedRuns)...)

Modified: pkg/cli/audit_cross_run.go — gh aw audit report integration

CrossRunAuditReport now includes a drain3_insights field ([]ObservabilityInsight). Phase 7 of buildCrossRunAuditReport calls buildDrain3InsightsFromCrossRunInputs to mine event templates across all runs in the report. Both markdown and pretty renderers include an "Agent Event Pattern Analysis (Drain3)" section with severity icons (🔴/🟠/🟡/ℹ), category labels, summaries, and evidence strings.

## Agent Event Pattern Analysis (Drain3)

### ℹ Log template patterns mined

**Category:** execution | **Severity:** info

Drain3 analysis identified 4 distinct event templates across 3 pipeline stages from 12 events.

_Evidence:_ `error=1 plan=1 tool_call=2`

New: cmd/agentdrain-demo/main.go

Runnable demo that shows the full pipeline: coordinator setup → pretraining → event ingestion → cluster/template/param/anomaly output → snapshot save/reload → post-restore inference.

Embedding default weights workflow

gh aw logs --train -c 100
cp .github/aw/logs/drain3_weights.json pkg/agentdrain/data/default_weights.json
make build   # re-embeds the trained weights into the binary

After this, all future gh aw audit and gh aw logs runs preload those templates into the Drain3 coordinator before processing new events, giving better anomaly detection from the first observation.

Testing

• 17 new unit tests in pkg/agentdrain/ — all pass
• 8 new tests in pkg/cli/drain3_integration_test.go — all pass
• 4 new tests in pkg/cli/drain3_train_test.go — all pass (including --train flag presence, JSON structure, stage key assertions)
• 5 new tests in pkg/cli/audit_cross_run_test.go — all pass (field-mapping assertions, insight field validation, markdown rendering)
• make lint — 0 issues
• make test-unit — all pre-existing failures are unrelated (require live GitHub API access)

Design decisions

• No external dependencies — pure standard library
• Structured event inputs — uses already-extracted audit fields (tool usage, MCP failures, etc.) rather than re-parsing raw logs, giving clean integration without I/O overhead
• Drain3 similarity — based on fraction of exact token matches at non-wildcard positions; length mismatch → similarity 0
• Thread-safe — sync.RWMutex throughout; Match uses RLock, Train uses full Lock
• Graceful degradation — unknown stages are skipped without error in the audit pipeline
• --train returns error — since the user explicitly requested training, failures are surfaced rather than silently warned

[pelikhan]

@copilot

add a flag "--train" to the "logs" command that runs the drain3 pre training to build the weights file which should be embedded to the compiler as default

[pelikhan]

@copilot

• integrate drain3 analysis into the "audit report" sub command report

[Copilot]

@copilot

add a flag "--train" to the "logs" command that runs the drain3 pre training to build the weights file which should be embedded to the compiler as default...

Done in commit d90b5f3.

--train flag added to gh aw logs:

gh aw logs --train                    # train on last 10 runs
gh aw logs --train -c 50              # train on up to 50 runs
gh aw logs my-workflow --train -o ./out  # custom output dir

How it works:

1. Downloads and processes runs as normal
2. Trains a Drain3 coordinator across all processed events (plan, tool_call, tool_result, retry, error, finish stages)
3. Writes drain3_weights.json to the output directory (pretty-printed JSON)
4. Prints instructions to embed as default:

   cp .github/aw/logs/drain3_weights.json pkg/agentdrain/data/default_weights.json
   make build
   

Embedding as default: The pkg/agentdrain/data/default_weights.json file is embedded via //go:embed in defaults.go. Once a trained weights file is committed there, all future gh aw audit and gh aw logs runs will preload those templates into the Drain3 coordinator before processing new events — giving better anomaly detection from the first run.

[Copilot]

@copilot

• integrate drain3 analysis into the "audit report" sub command report...

Done in commit 9809766.

Drain3 multi-run pattern analysis is now part of gh aw audit report (gh aw audit report --workflow my-workflow).

What was added:

• CrossRunAuditReport gets a new drain3_insights JSON field ([]ObservabilityInsight)
• Phase 7 in buildCrossRunAuditReport converts the existing crossRunInput slice to ProcessedRuns and runs buildDrain3InsightsMultiRun
• Both markdown (--format markdown) and pretty (--format pretty) renderers include an "Agent Event Pattern Analysis (Drain3)" section with severity icons (🔴/🟠/🟡/ℹ), category, summary, and evidence

Markdown output example:

## Agent Event Pattern Analysis (Drain3)

### ℹ Log template patterns mined

**Category:** execution | **Severity:** info

Drain3 analysis identified 4 distinct event templates across 3 pipeline stages from 12 events.

_Evidence:_ `error=1 plan=1 tool_call=2`

[pelikhan]

@copilot don't mention drain3 in report (internal detail)

[Copilot]

@copilot don't mention drain3 in report (internal detail)

Done in commit 2a9876a. All user-facing output now refers to "Agent Event Pattern Analysis" / "log pattern weights" without mentioning Drain3. The name remains only in internal identifiers (package, variable names, function names, debug logs).