Wire error categories into executor event reporting

[dejanzele]

What type of PR is this?

Feature (PR 2 of 4)

What this PR does / why we need it

Wires the classifier and FailureInfo into the executor's event reporting path.

• Constructs the Classifier from config at executor startup unconditionally (with no rules configured, it simply returns empty categories)
• Passes classifier to the pod issue handler and job state reporter
• Calls ExtractFailureInfo() + classifier.Classify() on every pod failure, attaching structured FailureInfo (exit code, termination message, categories, container name) to the Error events sent through Pulsar
• After this PR, every failed pod event flowing into Pulsar carries exit code, termination message, container name, and matched category names

Which issue(s) this PR fixes

Part of #4713 (Error Categorization)

Special notes for your reviewer

• This is PR 2 of 4: Proto + classifier (Add error categorization proto schema and executor classifier #4741) -> Wire into executor (this) -> Store in lookout DB + UI (Store FailureInfo in lookout DB and expose via API #4755) -> e2e tests (Add e2e test cases for error categorization #4760)
• Depends on Add error categorization proto schema and executor classifier #4741
• The classifier is always instantiated at startup, even with no rules configured - no conditional nil-checking needed at call sites
• Changes are in application.go (startup wiring), pod_issue_handler.go, job_state_reporter.go, reporter/event.go, cluster_allocation.go, and preempt_runs.go

[greptile-apps]

Greptile Summary

This PR (2 of 4 in the error categorization series) wires the Classifier and FailureInfo into the executor's event reporting path, so every failed pod event flowing into Pulsar now carries structured failure data (exit code, termination message, container name, and matched category names).

Key changes:

• Classifier is constructed unconditionally at executor startup from config.Application.ErrorCategories; with no rules configured it returns empty categories, so no nil-checks are required at call sites.
• CreateEventForCurrentState, CreateJobFailedEvent, and CreateSimpleJobFailedEvent gain new classifier/failureInfo parameters — all nil-safe via the Classify nil-receiver guard.
• ExtractFailureInfo + classifier.Classify are called in two places: CreateEventForCurrentState (for the PodFailed phase path) and handleNonRetryableJobIssue (for stuck/issue-handler path).
• Preempted and submission-failure events deliberately pass nil for FailureInfo.
• Tests cover nil classifier, classifier-with-rules, and the end-to-end issue-handler integration path.

Two minor concerns worth tracking toward the final PR in the series:

• ExtractFailureInfo always returns a non-nil proto even when no container exit data exists (e.g., eviction, deadline-exceeded), leaving ExitCode=0 and empty ContainerName — downstream consumers cannot distinguish "genuinely zero" from "not found."
• handleNonRetryableJobIssue attaches FailureInfo for every non-retryable issue type including ExternallyDeleted and ErrorDuringIssueHandling, where container-level data is typically absent.

Confidence Score: 5/5

Safe to merge — all remaining findings are P2 style/semantic concerns with no current runtime breakage.

The core wiring is correct and nil-safe throughout. The two findings are forward-looking quality concerns (zero-ExitCode ambiguity for eviction events, and FailureInfo attached to all non-retryable issue types regardless of relevance), neither of which causes broken behaviour in the current PR. Previous substantive concerns about PodCheckRetryable, preemption field divergence, and multi-container ExitCode correlation were already flagged in prior review threads. Test coverage for the new paths is solid.

internal/executor/reporter/event.go and internal/executor/service/pod_issue_handler.go carry the two P2 notes, but neither blocks merge.

Important Files Changed

Filename	Overview
internal/executor/application.go	Adds Classifier construction from config at startup and threads it into PodIssueHandler and JobStateReporter; straightforward wiring with fatal-on-config-error guard.
internal/executor/reporter/event.go	Adds classifier param to CreateEventForCurrentState and failureInfo param to CreateJobFailedEvent/CreateSimpleJobFailedEvent; nil-safe via classifier's nil-receiver guard, but always attaches non-nil FailureInfo even for eviction/deadline pods where no container exit code exists.
internal/executor/service/pod_issue_handler.go	Adds classifier field and wires ExtractFailureInfo+Classify into handleNonRetryableJobIssue; nil-safe but ExtractFailureInfo runs for all non-retryable issue types including eviction/external-deletion where container data may not be present.
internal/executor/service/job_state_reporter.go	Adds classifier field, passes it to CreateEventForCurrentState; straightforward and nil-safe delegation.
internal/executor/service/cluster_allocation.go	Updates call to CreateSimpleJobFailedEvent with explicit nil failureInfo for submission failures — deliberate, as these are not runtime pod failures.
internal/executor/job/processors/preempt_runs.go	Updates CreateSimpleJobFailedEvent call with explicit nil failureInfo for preempted runs — deliberate choice to not attach container failure detail for preemption events.
internal/executor/reporter/event_test.go	Adds coverage for nil classifier, non-nil classifier with exit-code matching, and asserts FailureInfo is always present on PodFailed events; good test hygiene.
internal/executor/service/pod_issue_handler_test.go	New integration-style test verifies classifier-matched categories appear on FailureInfo for stuck-terminating OOMKilled pods; well-structured and tests end-to-end path.

Sequence Diagram

sequenceDiagram
    participant App as application.go
    participant Clf as Classifier
    participant JSR as JobStateReporter
    participant PIH as PodIssueHandler
    participant Evt as reporter/event.go
    participant Util as util.ExtractFailureInfo
    participant Pulsar as Pulsar (EventSender)

    App->>Clf: NewClassifier(config.ErrorCategories)
    App->>PIH: NewPodIssuerHandler(..., classifier)
    App->>JSR: NewJobStateReporter(..., classifier)

    Note over JSR: Pod phase changes (Kubernetes watch)
    JSR->>Evt: CreateEventForCurrentState(pod, clusterId, classifier)
    Evt->>Clf: classifier.Classify(pod)
    Clf-->>Evt: []string categories
    Evt->>Util: ExtractFailureInfo(pod, categories)
    Util-->>Evt: *FailureInfo
    Evt-->>JSR: EventSequence (with FailureInfo on PodFailed)
    JSR->>Pulsar: Report(event)

    Note over PIH: Non-retryable issue detected
    PIH->>Clf: classifier.Classify(originalPodState)
    Clf-->>PIH: []string categories
    PIH->>Util: ExtractFailureInfo(originalPodState, categories)
    Util-->>PIH: *FailureInfo
    PIH->>Evt: CreateSimpleJobFailedEvent(..., failureInfo)
    Evt-->>PIH: EventSequence (with FailureInfo)
    PIH->>Pulsar: Report(event)

Loading

_{Reviews (25): Last reviewed commit: "Wire error categories into executor even..." | Re-trigger Greptile}

[dejanzele]

@greptile

[dejanzele]

@greptile

[dejanzele]

Updated the default FailureCondition from USER_ERROR to UNSPECIFIED in the upstream PR, so this concern is addressed now.

[dejanzele]

@greptileai