Feat/incremental delta processing

[ZohaibHassan16]

Description

This PR introduces Delta-Aware Pipelines. It allows the execution engine to dynamically intercept specific pipeline steps, compute the exact set-difference (added/removed triples) between two graph snapshots directly on the database backend, and pass only those changes downstream for validation or enrichment.

Type of Change

Bug fix New feature Breaking change Documentation update Performance improvement Code refactoring

Related Issues

Closes

#323

Changes Made

Storage & Metadata

Updated version_storage.py to track graph_uri explicitly. Added prune_versions to managers.py to clean up obsolete snapshots and optionally drop them from the DB to prevent storage bloat.

Native Delta Computation

Added compute_delta to triplet_store.py, utilizing SPARQL FILTER NOT EXISTS to push diff computation down to the database rather than loading full graphs into memory.

Pipeline Orchestration

Extended PipelineBuilder to support delta_mode, base_version_id, and target_version_id. Updated ExecutionEngine._execute_step to dynamically intercept delta steps, resolve URIs, compute the delta, and feed the diff payload to the handler.

Documentation

Added incremental processing guides, architectures, and pipeline configuration examples to change_management.md and pipeline.md.

Testing

Tested locally Added tests for new functionality Package builds successfully (python -m build)

Test Commands

# Verify the delta interception logic and logical equivalence
pytest tests/pipeline/test_pipeline_comprehensive.py::TestPipelineComprehensive::test_execution_engine_delta_mode -v

Documentation

Updated relevant documentation Added code examples if applicable Updated API reference if adding new APIs Updated cookbook if adding new examples No documentation changes needed

Breaking Changes

Status

No

Checklist

My code follows the project's style guidelines I have performed a self-review of my code I have commented my code, particularly in hard-to-understand areas My changes generate no new warnings (Note: Skipped some pre-existing flake8 warnings on unrelated lines to keep this PR's scope strict) Package builds successfully