#685 Fix when transformers and sinks are marked as not allowed to run in parallel have the behavior of transformations with self-dependencies.

[yruslan]

Closes #685

Summary by CodeRabbit

• New Features
  • Introduced job self-dependency detection. Task runner now allows non-dependent tasks to execute even after preceding task failures, improving pipeline resilience and efficiency.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Walkthrough

Introduces isSelfDependent as a new public API property in the Job trait and implementations. The task runner's failure-skipping logic is updated to only skip subsequent tasks on failure if the current job is self-dependent, rather than skipping all subsequent tasks when a job runs non-parallel.

Changes

Cohort / File(s)	Summary
Job trait and base implementation pramen/core/src/main/scala/za/co/absa/pramen/core/pipeline/Job.scala, pramen/core/src/main/scala/za/co/absa/pramen/core/pipeline/JobBase.scala	Added abstract member isSelfDependent: Boolean to Job trait; JobBase overrides with hasSelfDependencies property to expose self-dependency status in public API.
Task runner execution logic pramen/core/src/main/scala/za/co/absa/pramen/core/runner/task/TaskRunnerBase.scala	Modified runDependentTasks to conditionally skip subsequent tasks only when current task's job is self-dependent and a previous failure occurred; non-self-dependent jobs continue execution after failures.
Test infrastructure and coverage pramen/core/src/test/scala/za/co/absa/pramen/core/mocks/job/JobSpy.scala, pramen/core/src/test/scala/za/co/absa/pramen/core/tests/runner/task/TaskRunnerBaseSuite.scala	JobSpy updated with hasSelfDependencies constructor parameter; allowRunningTasksInParallel now depends on both parallel flag and self-dependency. New test case verifies sequential failure handling for self-dependent jobs.

Sequence Diagram

sequenceDiagram
    participant TaskRunner as Task Runner
    participant Task
    participant Job
    participant NextTask as Next Task

    TaskRunner->>Task: Execute task
    Task-->>TaskRunner: Fails
    TaskRunner->>Job: Check isSelfDependent
    
    alt Job is self-dependent
        Job-->>TaskRunner: true
        TaskRunner->>NextTask: Skip task
        Note over TaskRunner: Previous failure + self-dependent<br/>→ Stop execution chain
    else Job is not self-dependent
        Job-->>TaskRunner: false
        TaskRunner->>NextTask: Execute task
        Note over TaskRunner: Previous failure but not self-dependent<br/>→ Continue execution chain
    end

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 A hop, skip, and conditional bound,
Self-dependent jobs now check the ground,
When failures strike non-parallel ways,
The rest march on through brighter days,
Smart skipping blooms where it's truly due! 🌱

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: fixing behavior for non-parallel operations to distinguish self-dependency from parallel=false settings.
Linked Issues check	✅ Passed	The PR implements the fix for issue #685 by introducing isSelfDependent property and modifying failure-skipping logic to only skip subsequent tasks for genuinely self-dependent jobs, not just non-parallel ones.
Out of Scope Changes check	✅ Passed	All changes are directly related to fixing issue #685: adding isSelfDependent API, implementing distinction between self-dependent and non-parallel jobs, and updating test coverage accordingly.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@pramen/core/src/main/scala/za/co/absa/pramen/core/pipeline/Job.scala`:
- Line 45: The new abstract member isSelfDependent on trait Job breaks
downstream implementations; provide a compatibility-safe default by making
isSelfDependent a concrete method returning false in the Job trait (i.e., change
the declaration in Job so implementations keep working while allowing
overrides), and update any internal implementers that need true to explicitly
override isSelfDependent.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Avoid breaking external Job implementations with new abstract API.

Adding a new abstract member to a public trait is binary/source breaking for downstream implementers that aren’t updated/recompiled. Consider providing a safe default (e.g., false) or explicitly documenting the required migration in release notes. Line 45.

💡 Suggested compatibility-friendly default

-  def isSelfDependent: Boolean
+  def isSelfDependent: Boolean = false

To verify in-repo implementers:

#!/bin/bash
# Find Job implementations that may need the new member.
rg -nP --type=scala '\bextends\s+Job\b|\bwith\s+Job\b'

🤖 Prompt for AI Agents

In `@pramen/core/src/main/scala/za/co/absa/pramen/core/pipeline/Job.scala` at line
45, The new abstract member isSelfDependent on trait Job breaks downstream
implementations; provide a compatibility-safe default by making isSelfDependent
a concrete method returning false in the Job trait (i.e., change the declaration
in Job so implementations keep working while allowing overrides), and update any
internal implementers that need true to explicitly override isSelfDependent.

[github-actions]

Unit Test Coverage

Overall Project	84.2%	🍏
Files changed	81.25%	🍏

Module	Coverage
pramen:core Jacoco Report	86.2%	🍏

Files

Module	File	Coverage
pramen:core Jacoco Report	JobBase.scala	91.78% -0.25%	❌
	TaskRunnerBase.scala	82.73%	🍏