Skip to content

Commit 1355331

Browse files
Copilotpelikhan
Copilot
and
pelikhan
authored
feat: integrate log template mining into audit report and logs (#24328)
* feat: add pkg/agentdrain - Drain3-style log template mining package Agent-Logs-Url: https://github.com/github/gh-aw/sessions/850383e4-6ce1-4a3d-aa07-dae32343caa6 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * feat: integrate drain3 analysis into audit report and logs Agent-Logs-Url: https://github.com/github/gh-aw/sessions/850383e4-6ce1-4a3d-aa07-dae32343caa6 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * fix: remove incorrect build tags from non-test source files Agent-Logs-Url: https://github.com/github/gh-aw/sessions/850383e4-6ce1-4a3d-aa07-dae32343caa6 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * feat: add --train flag to logs command for drain3 weight pretraining Agent-Logs-Url: https://github.com/github/gh-aw/sessions/621cd144-30cc-44cd-9e7c-37361cee1b70 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * fix: address code review - bytes.TrimSpace, log pretty-print errors, return train error Agent-Logs-Url: https://github.com/github/gh-aw/sessions/621cd144-30cc-44cd-9e7c-37361cee1b70 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * feat: integrate drain3 analysis into audit report subcommand Agent-Logs-Url: https://github.com/github/gh-aw/sessions/1361e355-3eb5-4c65-9f64-ee483320bd65 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * fix: improve drain3 cross-run test assertions and doc comment Agent-Logs-Url: https://github.com/github/gh-aw/sessions/1361e355-3eb5-4c65-9f64-ee483320bd65 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * fix: remove Drain3 name from user-facing report output Agent-Logs-Url: https://github.com/github/gh-aw/sessions/89ebe149-2934-400a-a97e-a8f73ee6bbe4 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * chore: add daily Drain3 weight training workflow; delete agentdrain demo binary (#24344) * Initial plan * feat: add daily drain3 weight training workflow and delete demo binary Agent-Logs-Url: https://github.com/github/gh-aw/sessions/c5c57258-f0a1-4ade-8afc-9c4464b162cc Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> Co-authored-by: Peli de Halleux <pelikhan@users.noreply.github.com>
1 parent 72e24ce commit 1355331

30 files changed

Lines changed: 2608 additions & 8 deletions

‎.github/workflows/train-drain3-weights.yml‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
name: Train Log Pattern Weights
2+
3+
on:
4+
schedule:
5+
- cron: "0 4 * * *" # Daily at 04:00 UTC
6+
workflow_dispatch:
7+
8+
permissions: {}
9+
10+
jobs:
11+
train:
12+
name: Download logs and train drain3 weights
13+
runs-on: ubuntu-latest
14+
timeout-minutes: 30
15+
permissions:
16+
contents: write
17+
pull-requests: write
18+
steps:
19+
- name: Checkout code
20+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
21+
22+
- name: Set up Go
23+
uses: actions/setup-go@4dc6199c7b1a012772edbd06daecab0f50c9053c # v6
24+
with:
25+
go-version-file: go.mod
26+
cache: true
27+
28+
- name: Build gh-aw
29+
run: make build
30+
31+
- name: Download run logs and train weights
32+
env:
33+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
34+
run: |
35+
mkdir -p /tmp/drain3-logs
36+
./gh-aw logs --train --output /tmp/drain3-logs --count 50
37+
38+
- name: Copy trained weights to source tree
39+
run: |
40+
if [ -f /tmp/drain3-logs/drain3_weights.json ]; then
41+
cp /tmp/drain3-logs/drain3_weights.json pkg/agentdrain/data/default_weights.json
42+
echo "✅ Weights file updated successfully"
43+
else
44+
echo "⚠️ No drain3_weights.json produced – skipping PR creation"
45+
exit 0
46+
fi
47+
48+
- name: Check for changes
49+
id: check-changes
50+
run: |
51+
if git diff --quiet pkg/agentdrain/data/default_weights.json; then
52+
echo "changes=false" >> "$GITHUB_OUTPUT"
53+
echo "No changes to default_weights.json – weights are already up to date"
54+
else
55+
echo "changes=true" >> "$GITHUB_OUTPUT"
56+
echo "Changes detected in default_weights.json"
57+
fi
58+
59+
- name: Configure Git
60+
if: steps.check-changes.outputs.changes == 'true'
61+
run: |
62+
git config --global user.name "github-actions[bot]"
63+
git config --global user.email "github-actions[bot]@users.noreply.github.com"
64+
65+
- name: Create pull request with updated weights
66+
if: steps.check-changes.outputs.changes == 'true'
67+
env:
68+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
69+
run: |
70+
BRANCH_NAME="ci/train-drain3-weights-$(date +%Y%m%d)"
71+
72+
git checkout -b "$BRANCH_NAME"
73+
git add pkg/agentdrain/data/default_weights.json
74+
git commit -m "chore: update drain3 default weights from daily training run"
75+
76+
git push origin "$BRANCH_NAME"
77+
78+
gh pr create \
79+
--title "chore: update drain3 default log pattern weights" \
80+
--body "This pull request updates the default Drain3 log pattern weights (\`pkg/agentdrain/data/default_weights.json\`) by training on the most recent workflow run logs.
81+
82+
## What changed
83+
- Re-trained log template clusters from the latest run logs using \`gh aw logs --train\`
84+
- Copied resulting \`drain3_weights.json\` to the embedded defaults path
85+
86+
## How to verify
87+
1. Build the binary with \`make build\`
88+
2. Run \`gh aw audit\` or \`gh aw logs --train\` and confirm the anomaly analysis reflects the updated patterns
89+
90+
This PR was created automatically by the [train-drain3-weights](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) workflow." \
91+
--head "$BRANCH_NAME" \
92+
--base main

‎pkg/agentdrain/anomaly.go‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
package agentdrain
2+
3+
import "strings"
4+
5+
// AnomalyDetector evaluates match results and produces AnomalyReports.
6+
type AnomalyDetector struct {
7+
threshold float64
8+
rareThreshold int
9+
}
10+
11+
// NewAnomalyDetector creates an AnomalyDetector with the given thresholds.
12+
func NewAnomalyDetector(simThreshold float64, rareClusterThreshold int) *AnomalyDetector {
13+
return &AnomalyDetector{
14+
threshold: simThreshold,
15+
rareThreshold: rareClusterThreshold,
16+
}
17+
}
18+
19+
// Analyze produces an AnomalyReport for a match result.
20+
//
21+
// - isNew indicates the line created a brand-new cluster.
22+
// - cluster is the cluster that was matched or created.
23+
func (d *AnomalyDetector) Analyze(result *MatchResult, isNew bool, cluster *Cluster) *AnomalyReport {
24+
report := &AnomalyReport{
25+
IsNewTemplate: isNew,
26+
NewClusterCreated: isNew,
27+
}
28+
29+
if !isNew {
30+
report.LowSimilarity = result.Similarity < d.threshold
31+
}
32+
33+
if cluster != nil {
34+
report.RareCluster = cluster.Size <= d.rareThreshold
35+
}
36+
37+
// Weighted anomaly score.
38+
var score float64
39+
if report.IsNewTemplate {
40+
score += 1.0
41+
}
42+
if report.LowSimilarity {
43+
score += 0.7
44+
}
45+
if report.RareCluster {
46+
score += 0.3
47+
}
48+
// Normalize to [0, 1].
49+
const maxScore = 2.0
50+
if score > maxScore {
51+
score = maxScore
52+
}
53+
report.AnomalyScore = score / maxScore
54+
55+
report.Reason = buildReason(report)
56+
return report
57+
}
58+
59+
// buildReason constructs a human-readable summary of detected anomalies.
60+
func buildReason(r *AnomalyReport) string {
61+
var parts []string
62+
if r.IsNewTemplate {
63+
parts = append(parts, "new log template discovered")
64+
}
65+
if r.LowSimilarity {
66+
parts = append(parts, "low similarity to known template")
67+
}
68+
if r.RareCluster {
69+
parts = append(parts, "rare cluster (few observations)")
70+
}
71+
if len(parts) == 0 {
72+
return "no anomaly detected"
73+
}
74+
return strings.Join(parts, "; ")
75+
}

‎pkg/agentdrain/anomaly_test.go‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
//go:build !integration
2+
3+
package agentdrain
4+
5+
import (
6+
"testing"
7+
)
8+
9+
func TestAnomalyDetection_NewTemplate(t *testing.T) {
10+
d := NewAnomalyDetector(0.4, 2)
11+
c := &Cluster{ID: 1, Template: []string{"stage=plan"}, Size: 1}
12+
result := &MatchResult{ClusterID: 1, Similarity: 1.0}
13+
14+
report := d.Analyze(result, true, c)
15+
16+
if !report.IsNewTemplate {
17+
t.Error("expected IsNewTemplate=true")
18+
}
19+
if !report.NewClusterCreated {
20+
t.Error("expected NewClusterCreated=true")
21+
}
22+
if report.AnomalyScore <= 0 {
23+
t.Errorf("expected positive anomaly score for new template, got %v", report.AnomalyScore)
24+
}
25+
}
26+
27+
func TestAnomalyDetection_LowSimilarity(t *testing.T) {
28+
d := NewAnomalyDetector(0.4, 2)
29+
// Size=5 means not rare; not new.
30+
c := &Cluster{ID: 1, Template: []string{"a", "b", "c"}, Size: 5}
31+
result := &MatchResult{ClusterID: 1, Similarity: 0.2}
32+
33+
report := d.Analyze(result, false, c)
34+
35+
if !report.LowSimilarity {
36+
t.Error("expected LowSimilarity=true for similarity below threshold")
37+
}
38+
if report.IsNewTemplate {
39+
t.Error("expected IsNewTemplate=false")
40+
}
41+
if report.AnomalyScore <= 0 {
42+
t.Errorf("expected positive anomaly score, got %v", report.AnomalyScore)
43+
}
44+
}
45+
46+
func TestAnomalyDetection_RareCluster(t *testing.T) {
47+
d := NewAnomalyDetector(0.4, 2)
48+
c := &Cluster{ID: 1, Template: []string{"a"}, Size: 1}
49+
result := &MatchResult{ClusterID: 1, Similarity: 0.9}
50+
51+
report := d.Analyze(result, false, c)
52+
53+
if !report.RareCluster {
54+
t.Error("expected RareCluster=true for size=1 with rareThreshold=2")
55+
}
56+
if report.AnomalyScore <= 0 {
57+
t.Errorf("expected positive anomaly score, got %v", report.AnomalyScore)
58+
}
59+
}
60+
61+
func TestAnomalyDetection_Normal(t *testing.T) {
62+
d := NewAnomalyDetector(0.4, 2)
63+
// High size, high similarity, not new.
64+
c := &Cluster{ID: 1, Template: []string{"a", "b"}, Size: 100}
65+
result := &MatchResult{ClusterID: 1, Similarity: 0.9}
66+
67+
report := d.Analyze(result, false, c)
68+
69+
if report.IsNewTemplate {
70+
t.Error("expected IsNewTemplate=false")
71+
}
72+
if report.LowSimilarity {
73+
t.Error("expected LowSimilarity=false")
74+
}
75+
if report.RareCluster {
76+
t.Error("expected RareCluster=false")
77+
}
78+
if report.AnomalyScore > 0 {
79+
t.Errorf("expected zero anomaly score for normal event, got %v", report.AnomalyScore)
80+
}
81+
if report.Reason != "no anomaly detected" {
82+
t.Errorf("expected 'no anomaly detected', got %q", report.Reason)
83+
}
84+
}
85+
86+
func TestAnalyzeEvent(t *testing.T) {
87+
cfg := DefaultConfig()
88+
m, err := NewMiner(cfg)
89+
if err != nil {
90+
t.Fatalf("NewMiner: %v", err)
91+
}
92+
93+
// First occurrence → new template.
94+
evt := AgentEvent{
95+
Stage: "plan",
96+
Fields: map[string]string{"action": "start", "model": "gpt-4"},
97+
}
98+
result, report, err := m.AnalyzeEvent(evt)
99+
if err != nil {
100+
t.Fatalf("AnalyzeEvent: %v", err)
101+
}
102+
if result == nil {
103+
t.Fatal("AnalyzeEvent: expected non-nil result")
104+
}
105+
if report == nil {
106+
t.Fatal("AnalyzeEvent: expected non-nil report")
107+
}
108+
if !report.IsNewTemplate {
109+
t.Error("first event should be a new template")
110+
}
111+
112+
// Second occurrence of the same event → not new.
113+
result2, report2, err := m.AnalyzeEvent(evt)
114+
if err != nil {
115+
t.Fatalf("AnalyzeEvent (second): %v", err)
116+
}
117+
if result2 == nil || report2 == nil {
118+
t.Fatal("AnalyzeEvent (second): expected non-nil results")
119+
}
120+
if report2.IsNewTemplate {
121+
t.Error("second identical event should not be a new template")
122+
}
123+
}

‎pkg/agentdrain/cluster.go‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
package agentdrain
2+
3+
// clusterStore manages the set of known log template clusters.
4+
type clusterStore struct {
5+
clusters map[int]*Cluster
6+
nextID int
7+
}
8+
9+
func newClusterStore() *clusterStore {
10+
return &clusterStore{
11+
clusters: make(map[int]*Cluster),
12+
nextID: 1,
13+
}
14+
}
15+
16+
// add creates a new Cluster for the given template and returns a pointer to it.
17+
func (s *clusterStore) add(template []string, stage string) *Cluster {
18+
id := s.nextID
19+
s.nextID++
20+
tmpl := make([]string, len(template))
21+
copy(tmpl, template)
22+
c := &Cluster{
23+
ID: id,
24+
Template: tmpl,
25+
Size: 1,
26+
Stage: stage,
27+
}
28+
s.clusters[id] = c
29+
return c
30+
}
31+
32+
// get retrieves a cluster by ID.
33+
func (s *clusterStore) get(id int) (*Cluster, bool) {
34+
c, ok := s.clusters[id]
35+
return c, ok
36+
}
37+
38+
// all returns a snapshot of all clusters as a value slice.
39+
func (s *clusterStore) all() []Cluster {
40+
out := make([]Cluster, 0, len(s.clusters))
41+
for _, c := range s.clusters {
42+
out = append(out, *c)
43+
}
44+
return out
45+
}
46+
47+
// computeSimilarity returns the fraction of positions where tokens a and b
48+
// match exactly, considering only positions that are not paramToken in a.
49+
// Returns 0 when the slices have different lengths.
50+
func computeSimilarity(a, b []string, paramToken string) float64 {
51+
if len(a) != len(b) {
52+
return 0
53+
}
54+
nonParam := 0
55+
matches := 0
56+
for i, tok := range a {
57+
if tok == paramToken {
58+
continue
59+
}
60+
nonParam++
61+
if tok == b[i] {
62+
matches++
63+
}
64+
}
65+
if nonParam == 0 {
66+
// All positions are wildcards – treat as a perfect structural match.
67+
return 1.0
68+
}
69+
return float64(matches) / float64(nonParam)
70+
}
71+
72+
// mergeTemplate produces a new template by replacing positions where the two
73+
// token slices differ with paramToken. Positions where either token already is
74+
// paramToken also become paramToken.
75+
func mergeTemplate(existing, incoming []string, paramToken string) []string {
76+
if len(existing) != len(incoming) {
77+
return existing
78+
}
79+
merged := make([]string, len(existing))
80+
for i, tok := range existing {
81+
if tok == paramToken || incoming[i] == paramToken || tok != incoming[i] {
82+
merged[i] = paramToken
83+
} else {
84+
merged[i] = tok
85+
}
86+
}
87+
return merged
88+
}

0 commit comments

Comments
 (0)