diff --git a/docs/context_current_pipeline_appendix.md b/docs/context_current_pipeline_appendix.md new file mode 100644 index 0000000..e3c97da --- /dev/null +++ b/docs/context_current_pipeline_appendix.md @@ -0,0 +1,223 @@ +# Pipeline Context: Supplementary Analysis + +This document contains implementation details and reference material that supplement the use cases in [context_use_cases_current_pipeline.md](context_use_cases_current_pipeline.md). These sections were separated to keep the use-case document focused on requirements. + +--- + +## Implementation Notes by Use Case + +The following implementation notes describe how each use case is realized in the current pipeline codebase. They were separated from the use-case definitions to keep the requirements document focused on requirements. + +### UC-01 — Load, Update, and Provide Access to Observation Metadata + +**Implementation notes** — `context.observing_run` holds the observation metadata and is the most frequently queried attribute of the context: + +- `context.observing_run.get_ms(name=vis)` — resolve an MS by filename +- `context.observing_run.measurement_sets` — all registered MS objects +- `context.observing_run.get_measurement_sets_of_type(dtypes)` — filter by data type (RAW, REGCAL_CONTLINE_ALL, BASELINED, etc.) +- `context.observing_run.virtual2real_spw_id(vspw, ms)` / `real2virtual_spw_id(...)` — translate between abstract pipeline SPW IDs and CASA-native IDs +- `context.observing_run.virtual_science_spw_ids` — virtual SPW catalog +- `context.observing_run.ms_reduction_group` — per-group reduction metadata (single-dish) +- Provenance attributes: `start_datetime`, `end_datetime`, `project_ids`, `schedblock_ids`, `execblock_ids`, `observers` + +The MS objects stored by `context.observing_run` carry information about scans, fields, SPWs, antennas, reference antenna ordering, etc. Tasks read per-MS state like `ms.reference_antenna`, `ms.session`, `ms.start_time`, `ms.origin_ms`. + +For the single-dish pipeline, this use case also includes per-MS `DataTable` products referenced through `context.observing_run.ms_datatable_name`. These are not just raw imported metadata tables: they persist row-level metadata and derived quantities used by downstream SD tasks. During SD import, the reader populates `DataTable` columns such as `RA`, `DEC`, `AZ`, `EL`, `SHIFT_RA`, `SHIFT_DEC`, `OFS_RA`, and `OFS_DEC`, including coordinate conversions into the pipeline's chosen celestial frame (for example ICRS) so later imaging, gridding, plotting, and QA code can reuse those values efficiently. + +--- + +### UC-02 — Store and Provide Project-Level Metadata + +**Implementation notes** — project metadata is set during initialization or import, is not modified after import, and is read many times: + +- `context.project_summary = project.ProjectSummary(...)` — set by `executeppr()` / `executevlappr()` +- `context.project_structure = project.ProjectStructure(...)` — set by PPR executors +- `context.project_performance_parameters` — performance parameters from the PPR +- `context.set_state('ProjectStructure', 'recipe_name', value)` — used by `recipereducer.reduce()` and SD heuristics +- `context.processing_intents` — set by `Pipeline` during initialization + +--- + +### UC-03 — Register, Query, and Update Calibration State + +**Implementation notes** — `context.callibrary` is the primary cross-stage communication channel for calibration workflows: + +- **Write:** `context.callibrary.add(calto, calfrom)` — register a calibration application (cal table + target selection); `context.callibrary.unregister_calibrations(matcher)` — remove by predicate +- **Read:** `context.callibrary.active.get_caltable(caltypes=...)` — list active cal tables; `context.callibrary.get_calstate(calto)` — get full application state for a target selection +- Backed by `CalApplication` → `CalTo` / `CalFrom` objects with interval trees for efficient matching. + +--- + +### UC-04 — Manage Imaging State + +**Implementation notes** — imaging state is stored as ad-hoc attributes on the context object with no formal schema. Defensive `hasattr()` checks appear throughout the code to guard against attributes that may not yet exist: + +| Attribute | Written by | Read by | +|---|---|---| +| `clean_list_pending` | `editimlist`, `makeimlist`, `findcont`, `makeimages` | `findcont`, `transformimagedata`, `makeimages`, `vlassmasking` | +| `clean_list_info` | `makeimlist`, `makeimages` | `makeimages` | +| `imaging_mode` | `editimlist` | `makermsimages`, `makecutoutimages`, `makeimages`, VLASS export/display code | +| `imaging_parameters` | `imageprecheck` | `tclean`, `checkproductsize`, `makeimlist`, heuristics | +| `synthesized_beams` | `imageprecheck`, `tclean`, `checkproductsize`, `makeimlist`, `makeimages` | `imageprecheck`, `editimlist`, `tclean`, `uvcontsub`, `checkproductsize`, heuristics | +| `size_mitigation_parameters` | `checkproductsize` | downstream stages | +| `selfcal_targets` | `selfcal` | `makeimlist` | +| `selfcal_resources` | `selfcal` | `exportdata` | + +--- + +### UC-05 — Register and Query Produced Image Products + +**Implementation notes** — image libraries provide typed registries: + +- `context.sciimlist` — science images +- `context.calimlist` — calibrator images +- `context.rmsimlist` — RMS images +- `context.subimlist` — sub-product images (cutouts, cubes) + +--- + +### UC-06 — Track Current Execution Progress + +**Implementation notes:** + +- `context.stage`, `context.task_counter`, `context.subtask_counter` track progress + +--- + +### UC-07 — Preserve Per-Stage Execution Record + +**Implementation notes:** + +- `context.results` holds an ordered list of `ResultsProxy` objects which are proxied to disk to bound memory +- Timetracker integration provides per-stage timing data +- Results proxies store basenames for portability + +--- + +### UC-08 — Propagate Task Outputs to Downstream Tasks + +**Implementation notes** — the intended primary mechanism in the current pipeline is immediate propagation through context state updated during result acceptance. Over time, some workflows also came to inspect recorded results directly. Both patterns exist in the codebase, but the second should be understood as an accreted pattern rather than the original design intent. + +This use case is also a concrete example of context creep caused by weakly enforced contracts: the intended contract was that downstream tasks would consume explicitly merged shared state, but later code sometimes reached into `context.results` directly when that contract was not maintained consistently. + +1. **Immediate state propagation** — `Results.merge_with_context(context)` updates calibration library, image libraries, and dedicated context attributes such as `clean_list_pending`, `clean_list_info`, `synthesized_beams`, `size_mitigation_parameters`, `selfcal_targets`, and `selfcal_resources` so later tasks can access the current processing state directly without parsing another task's results object. +2. **Recorded-result inspection** — some tasks read `context.results` to find outputs from earlier stages when those outputs are needed from the recorded results rather than from merged shared state. This pattern introduces coupling to recipe order or to another task's result class structure. For example: + - VLA tasks compute `stage_number` from `context.results[-1].read().stage_number + 1` + - `vlassmasking` iterates `context.results[::-1]` to find the latest `MakeImagesResult` + - Export/AQUA code reads `context.results[0]` and `context.results[-1]` for timestamps + +--- + +### UC-09 — Provide a Transient Intra-Stage Workspace + +**Implementation notes** — the current framework implements this behavior in `pipeline/infrastructure/basetask.py`: + +- `StandardTaskTemplate.execute()` replaces `self.inputs` with a pickled copy of the original inputs, including the context, before task logic runs, and restores the original inputs in `finally` +- Child tasks therefore execute against a duplicated context that may be mutated freely during `prepare()` / `analyse()` +- `Executor.execute(job, merge=True)` commits a child result by calling `result.accept(self._context)`; with `merge=False`, the child task may still be run and inspected without committing its state +- This makes it possible for aggregate tasks to try tentative calibration paths or other destructive edits inside a stage and keep only the results they explicitly accept +- The rollback mechanism is in-memory copy/restore of task inputs and context; it is distinct from explicit session save/resume workflows + +--- + +### UC-10 — Support Multiple Orchestration Drivers + +**Implementation notes** — multiple entry points converge on the same task execution path: + +- **Task-driven**: direct task calls via CLI wrappers in `pipeline/h/cli/` +- **Command-list-driven**: PPR and XML procedure commands via `executeppr.py` / `executevlappr.py` and `recipereducer.py` + +They differ in how inputs are specified, how session paths are selected, and how resume is initiated, but the persisted context is the same. + +--- + +### UC-11 — Save and Restore a Processing Session + +**Implementation notes:** + +- `h_save()` pickles the whole context to `.context` +- `h_resume(filename)` loads a `.context` file, defaulting to the most recent context file available if `filename` is `None` or `last` is used. +- Per-stage results are proxied to disk (`saved_state/result-stageX.pickle`) to keep the in-memory context smaller +- Used by driver-managed breakpoint/resume (`executeppr(..., bpaction='resume')`) and developer debugging workflows + +--- + +### UC-12 — Provide State to Parallel Workers + +**Implementation notes** — `pipeline/infrastructure/mpihelpers.py`, class `Tier0PipelineTask`: + +1. The MPI client saves the context to disk as a pickle: `context.save(path)`. +2. Task arguments are also pickled to disk alongside the context. +3. On the server, `get_executable()` loads the context, modifies `context.logs['casa_commands']` to a server-local temp path, creates the task's `Inputs(context, **task_args)`, then executes the task. +4. For `Tier0JobRequest` (lower-level distribution), the executor is shallow-copied *excluding* the context reference to stay within the pipeline-enforced MPI buffer limit (100 MiB). Comments in the code note CASA's higher native limit (~150 MiB; see PIPE-1337 / CAS-13656). + +--- + +### UC-14 — Provide Read-Only State for Reporting + +**Implementation notes** — `WebLogGenerator.render(context)` in `pipeline/infrastructure/renderer/htmlrenderer.py`: + +- `WebLogGenerator.render(context)` explicitly does `context.results = [proxy.read() for proxy in context.results]` once before the renderer loop, so individual renderers iterate fully unpickled result objects rather than calling `read()` themselves +- Reads `context.report_dir`, `context.output_dir` — filesystem layout +- Reads `context.observing_run.*` — MS metadata, scheduling blocks, execution blocks, observers, project IDs, start/end times +- Reads `context.project_summary.telescope` — to determine telescope-specific page layouts (ALMA vs VLA vs NRO) +- Reads `context.project_structure.*` — OUS IDs, PPR file, recipe name +- The larger renderer stack, including the Mako templates under `pipeline/infrastructure/renderer/templates/`, reads `context.logs['casa_commands']` and related log references when generating weblog links + +--- + +### UC-15 — Support QA Evaluation and Store Quality Assessments + +**Implementation notes** — after `merge_with_context()`, `accept()` triggers `pipelineqa.qa_registry.do_qa(context, result)`: + +- QA handlers implement `QAPlugin.handle(context, result)` +- The context provides inputs to QA evaluation: + - Most handlers call `context.observing_run.get_ms(vis)` to look up metadata for scoring (antenna count, channel count, SPW properties, field intents) + - Some handlers check `context.imaging_mode` to branch on VLASS-specific scoring + - Others check things in `context.observing_run`, `context.project_structure`, or the callibrary (`context.callibrary`) +- Scores are appended to `result.qa.pool`, so the scores are stored on the results rather than directly on the context. This also keeps detailed QA collections scoped to the stage result that produced them; in current code, a `QAScorePool` can hold many `QAScore` objects, and each score may carry fine-grained `applies_to` selections (e.g. vis, field, SPW, antenna, polarization), so the per-result pool can become fairly large for detailed assessments. + +QA handlers write scores to `result.qa.pool` and do not modify the shared context directly. + +--- + +### UC-17 — Manage Telescope- and Array-Specific State + +**Implementation notes** — the current codebase shows at least two different forms of telescope-/array-specific state. + +One is a VLA-specific sub-context (`context.evla`) which is created during `hifv_importdata` and is updated by several subsequent tasks. Functionally, it provides a way to store observation metadata and pass state between tasks under `context.evla` rather than using the top-level context directly or other context objects (e.g. the domain objects). `context.evla` is an untyped, dictionary-of-dictionaries sidecar dynamically attached to the top-level context with no schema, no type annotations, and no declaration in `Context.__init__`. + +`context.evla` is a `collections.defaultdict(dict)`, keyed as `context.evla['msinfo'][ms_name].`: + +- **Written by:** `hifv_importdata` (creates + initializes), `testBPdcals` (gain intervals, ignorerefant), `fluxscale/solint`, `fluxboot` +- **Read by:** Most VLA calibration tasks and heuristics +- Accessed fields include: `gain_solint1`, `gain_solint2`, `setjy_results`, `ignorerefant`, various `*_field_select_string` / `*_scan_select_string` values, `fluxscale_sources`, `spindex_results`, and many more + +Another is ALMA TP / single-dish state, which is array-specific rather than telescope-wide and is carried mainly through SD-specific structures under `context.observing_run`, such as `ms_datatable_name`, `ms_reduction_group`, and `org_directions`, plus the per-MS `DataTable` products referenced from that state. This is a useful reminder that array-specific extensions do not always appear as a single sidecar object like `context.evla`; they may instead live in domain-model extensions and array-specific cached metadata products. + +--- + +## Key Implementation References + +- `Context` / `Pipeline`: `pipeline/infrastructure/launcher.py` +- CLI lifecycle tasks: `pipeline/h/cli/h_init.py`, `pipeline/h/cli/h_save.py`, `pipeline/h/cli/h_resume.py` +- Task dispatch & result acceptance: `pipeline/h/cli/utils.py`, `pipeline/infrastructure/basetask.py` +- PPR-driven execution loops: + - ALMA: `pipeline/infrastructure/executeppr.py` (used by `pipeline/runpipeline.py`) + - VLA: `pipeline/infrastructure/executevlappr.py` (used by `pipeline/runvlapipeline.py`) +- Direct XML procedure execution: `pipeline/recipereducer.py` +- MPI distribution: `pipeline/infrastructure/mpihelpers.py` +- QA framework: `pipeline/infrastructure/pipelineqa.py`, `pipeline/qa/` +- Weblog renderer: `pipeline/infrastructure/renderer/htmlrenderer.py` + +--- + +## Context Lifecycle + +The canonical flow through the context is: + +1. **Create session** — `h_init()` constructs a `launcher.Pipeline(...)` and returns a new `Context`. In PPR-driven execution, `executeppr()` or `executevlappr()` also populates project metadata at this point. +2. **Load data** — Import tasks (`h*_importdata`) attach datasets to the context's domain model (`context.observing_run`, measurement sets, scans, SPWs, etc.). +3. **Execute tasks** — Tasks execute against the in-memory context and return a `Results` object. After each task, `Results.accept(context)` records the outcome and mutates shared state. +4. **Accept results** — Inside `accept()`, results are merged via `Results.merge_with_context(context)`. A `ResultsProxy` is pickled to disk per-stage to keep the in-memory context bounded. The weblog is typically rendered after each top-level stage. +5. **Save / resume** — `h_save()` pickles the context; `h_resume(filename='last')` restores it. Driver-managed breakpoints and developer debugging workflows rely on this cycle. diff --git a/docs/context_gap_use_cases.md b/docs/context_gap_use_cases.md new file mode 100644 index 0000000..6ae6726 --- /dev/null +++ b/docs/context_gap_use_cases.md @@ -0,0 +1,88 @@ +## Missing capabilities (GAPs) and implications for RADPS context design + +This document records capabilities the current pipeline context design cannot yet support. Not every item below is strictly a "context" feature, but each implies changes to context responsibilities, schema, or interfaces. The gaps are enumerated as GAP-01 through GAP-08. A separate, more exhaustive gap analysis mapping these use cases to RADPS requirements is recommended. + +## High-level gap list + +- GAP-01: Concurrent / overlapping execution — requires snapshot isolation, transactional merges, partition-scoped writes, and conflict detection. +- GAP-02: Distributed execution without a shared filesystem — requires artifact references decoupled from POSIX paths and a context that serves as the system-of-record. +- GAP-03: Provenance and reproducibility — requires immutable per-attempt records, input hashing, and lineage capture. +- GAP-04: Partial re-execution / targeted rerun — requires explicit dependency tracking and invalidation semantics at the context level. +- GAP-05: External system integration — requires stable identifiers, event subscriptions/webhooks, and exportable summaries/manifests. +- GAP-06: Programming-language / client-framework access — requires a language-neutral contract and stable middleware/API layer with extensible value types so clients in any language can access context state without coupling to the storage representation. +- GAP-07: Streaming / incremental processing — requires versioned dataset registration and versioned results/artifacts. +- GAP-08: Cross-MS matching and heterogeneous dataset coordination — requires flexible SPW matching semantics (exact and partial/overlap) and data-type tracking across measurement sets, rather than the current single-master-MS assumption. + +## Detailed use cases + +### GAP-01 — Concurrent execution of independent work + +| | | +|-------|---------| +| **Actor(s)** | Workflow orchestration layer, parallel task scheduler | +| **Summary** | The context must support concurrent execution at multiple granularities (stage-level and within-stage parallelism) while preventing inconsistent processing state. This differs from the current parallel-worker pattern, which waits for all work to finish before proceeding. | +| **Invariant** | Independent tasks may run concurrently but must not produce conflicting state. | +| **Postconditions** | Results from concurrent tasks are fully and consistently incorporated into processing state before any dependent work begins. | +| **RADPS requirements** | CSS9017, CSS9063 | + +### GAP-02 — Distributed execution without a shared filesystem + +| | | +|-------|---------| +| **Actor(s)** | Workflow orchestration layer, distributed workers | +| **Summary** | Execution must be possible across nodes that do not share a filesystem. Artifacts, datasets, and processing state must be addressable and accessible without relying on local POSIX paths. | +| **Postconditions** | Processing completes across distributed nodes with context-hosted references providing the necessary artifact access. | +| **RADPS requirements** | CSS9002, CSS9030 | + +### GAP-03 — Provenance and reproducibility + +| | | +|-------|---------| +| **Actor(s)** | Pipeline operator, auditor, reproducibility tooling | +| **Summary** | The context must record sufficient provenance — software versions, exact input identities/hashes, task parameters, and per-stage state — to enable precise reproduction and audit of past runs. | +| **Postconditions** | Any past processing step can be reproduced or audited using the recorded provenance chain. | +| **RADPS requirements** | ALMA-TR103, ALMA-TR104, ALMA-TR105 | + +### GAP-04 — Partial re-execution / targeted stage re-run + +| | | +|-------|---------| +| **Actor(s)** | Pipeline operator, developer, workflow engine | +| **Summary** | The context must support selectively re-running one or more mid-pipeline stages with new parameters while preserving unaffected stages. Downstream stages that depend on changed outputs must be invalidated or recomputed. | +| **Postconditions** | Processing state reflects the re-run outcomes; affected downstream stages are invalidated or updated; unaffected stages remain intact. | +| **RADPS requirements** | CSS9038 | + +### GAP-05 — External system integration (archive, scheduling, QA dashboards) + +| | | +|-------|---------| +| **Actor(s)** | QA dashboards, monitoring tools, archive ingest systems, schedulers | +| **Summary** | External systems need timely access to processing state (current stage, processing time, QA results, lifecycle events) without waiting for offline product files. The context should expose the necessary state via queryable interfaces or event feeds. | +| **Invariant** | External consumers can access the processing state they require while it remains current. | +| **Postconditions** | External systems can track processing progress and lifecycle transitions in near real time. | +| **RADPS requirements** | CSS9046, CSS9047, CSS9048, CSS9049, CSS9050, CSS9056 | + +### GAP-06 — Programming-Language / Client-Framework Access to Context + +| | | +|-------|---------| +| **Actor(s)** | Non-Python clients (C++, Julia, JavaScript dashboards), external tools | +| **Summary** | Expose context state through a programming-language-neutral contract and a stable middleware/API layer (e.g., REST/gRPC over a typed schema such as Protocol Buffers, JSON-Schema, or Arrow) so local language bindings do not need to know how the context is physically stored. The contract should provide a small set of standard extensible value types (scalars, lists, maps/dictionaries, records/structs) alongside typed core records so developers can add new items quickly without redefining the storage model for each client. | +| **Postcondition** | Clients in any supported programming language can query context state and artifacts through a stable typed API without coupling themselves to the underlying storage representation. | + +### GAP-07 — Streaming / incremental processing + +| | | +|-------|---------| +| **Actor(s)** | Data ingest systems, workflow engine, incremental processing tasks | +| **Summary** | Support incremental dataset registration (adding new scans or execution blocks to a live session), incremental detection and processing of new data, and versioned results so re-runs produce new versions rather than overwriting. | +| **Postconditions** | New data may be incorporated into an active session and processed incrementally without restarting the pipeline from scratch. | + +### GAP-08 — Cross-MS matching and heterogeneous dataset coordination + +| | | +|-------|---------| +| **Actor(s)** | Data import tasks, calibration tasks, imaging tasks, heuristics | +| **Summary** | The current design provides limited support for heterogeneous multi-MS datasets through virtual SPW translation and per-MS data-column tracking, but many workflows still rely on a single reference-MS or master-MS model and do not expose general cross-MS matching semantics. The context must instead support heterogeneous multi-MS scenarios by providing: (1) cross-MS SPW matching with distinct semantics for exact matching (required by calibration tasks) and partial/overlap matching (required for imaging tasks that can combine overlapping spectral windows); and (2) data-type and column tracking across multiple MSes without assuming a shared layout. Because the current virtual-SPW translation mechanism is tightly coupled to the single-master-MS assumption, a fresh design is preferable to extending it. | +| **Invariant** | SPW identity and data-column state are queryable across all registered datasets, regardless of whether those datasets share native numbering or column layout. | +| **Postconditions** | Calibration and imaging tasks can look up applicable SPWs and data columns across an arbitrary collection of heterogeneous MSes using the appropriate matching semantics for their use. | diff --git a/docs/context_use_cases_current_pipeline.md b/docs/context_use_cases_current_pipeline.md new file mode 100644 index 0000000..b5a5a02 --- /dev/null +++ b/docs/context_use_cases_current_pipeline.md @@ -0,0 +1,202 @@ +# Pipeline Context Use Cases + +## Overview +The pipeline context is the central state object used for a pipeline execution. It carries observation data, calibration state, imaging state, execution history and state, project metadata, and serves as the primary communication channel between pipeline stages. + +This document catalogues the use cases of the current pipeline context as determined by examination of the codebase. The goal is to inform the design of a system serving a similar role to the current pipeline context for RADPS. + +For additional details about the current implementation and reference material, see [Supplementary Analysis](context_current_pipeline_appendix.md). + +--- + +## Use Cases + +Each use case describes the required capabilities of the context system and the interactions through which those capabilities are exercised. They are written to be implementation-neutral — the goal is to capture what the context must do, not how the current pipeline implementation achieves it. For pipeline-specific implementation details by use case, see [Implementation Notes by Use Case](context_current_pipeline_appendix.md#implementation-notes-by-use-case) in the appendix. + +The following fields are used in each use case: + +- **Actor(s):** The human or system role that directly creates, updates, consumes, or inspects the context state described by the use case. Actors are role categories, not specific task names or current implementations. +- **Summary:** What the context must do to satisfy the use case. +- **Invariant:** A condition that must always be true while the system is operating. Present only where a meaningful invariant exists. +- **Postcondition:** A condition that must be true after a specific operation completes. Present only where a meaningful postcondition exists. + +--- + +### UC-01 — Load, Update, and Provide Access to Observation Metadata + +| | | +|-------|---------| +| **Actor(s)** | Data import task, any downstream task, heuristics, renderers, QA handlers | +| **Summary** | The context must load observation metadata (datasets, spectral windows, fields, antennas, scans, time ranges), make it queryable by all subsequent processing steps, and allow downstream tasks to update it as processing progresses (e.g., registering new derived datasets, data column and type changes, reference antenna selection). It must also be able to hold derived or cached metadata products created during import when later stages rely on them for efficiency rather than recomputing them from the raw measurement set, and it must provide a unified identifier scheme when multiple datasets use different native numbering. | +| **Invariant** | All registered datasets remain queryable and updatable for the lifetime of the session without repeating the import process. | + +--- + +### UC-02 — Store and Provide Project-Level Metadata + +| | | +|-------|---------| +| **Actor(s)** | Initialization, any task, report generators | +| **Summary** | The context must store project-level metadata (proposal code, PI, telescope, desired sensitivities, processing recipe) and make it available to all components, e.g., for decision-making in heuristics and to label outputs in reports. | +| **Invariant** | Project metadata is available for the lifetime of the processing session. | + +--- + +### UC-03 — Register, Query, and Update Calibration State + +| | | +|-------|---------| +| **Actor(s)** | Calibration tasks, export tasks, restore tasks, report generators | +| **Summary** | The context must allow calibration tasks to register and update solutions (indexed by data selection: field, spectral window, antenna, intent, time interval), and allow downstream tasks to query for all calibrations applicable to a given data selection. It must distinguish between calibrations pending application and those already applied. Registration must support registering multiple calibrations atomically as part of a single operation. | +| **Invariant** | Calibration state is queryable and correctly scoped to data selections, and can be updated as processing progresses. | + +--- + +### UC-04 — Manage Imaging State + +| | | +|-------|---------| +| **Actor(s)** | Imaging tasks, downstream heuristics, and export tasks | +| **Summary** | The context must allow imaging state — target lists, imaging parameters, masks, thresholds, and sensitivity estimates — to be computed by one step and read or refined by later steps. Multiple steps may contribute to a progressively refined imaging configuration. | +| **Invariant** | Imaging state reflects contributions from all completed imaging-related stages, and is available for reading or refinement by subsequent stages. | + +--- + +### UC-05 — Register and Query Produced Image Products + +| | | +|-------|---------| +| **Actor(s)** | Imaging tasks, export tasks, report generators | +| **Summary** | The context must maintain typed registries of produced image products with add/query semantics. Later tasks must be able to discover previously produced science, calibrator, RMS, and sub-product images through these registries. | +| **Invariant** | Produced image products are registered by type and remain queryable for downstream processing, reporting, and export. | + +--- + +### UC-06 — Track Current Execution Progress + +| | | +|-------|---------| +| **Actor(s)** | Workflow orchestration layer, tasks, pipeline operators | +| **Summary** | The context must track which processing stage is currently executing and maintain a stable, ordered record of completed stages. Stage identity and ordering must remain coherent across session saves and resumes. | +| **Invariant** | The currently executing stage is identifiable and completed stages are recorded in stable order. | + +___ + +### UC-07 — Preserve Per-Stage Execution Record + +| | | +|-------|---------| +| **Actor(s)** | Report generators, pipeline operators, workflow orchestration layer | +| **Summary** | The context must preserve a complete execution record for each completed stage, including timing, traceback information, outcomes, and the arguments used to invoke it. This record must support reporting, post-mortem diagnosis of failures, and resumption after interruption. | +| **Invariant** | Each completed stage retains its full execution record identity, outcome, timing, traceback, and invocation arguments for the lifetime of the session. | + +--- + +### UC-08 — Propagate Task Outputs to Downstream Tasks + +| | | +|-------|---------| +| **Actor(s)** | Any task producing output, downstream tasks | +| **Summary** | When a task produces outputs that change the processing state (e.g., new calibrations, updated flag summaries, image products, revised parameters), the context must provide a mechanism for those outputs to become available to subsequent processing steps before they execute. UC-03, UC-04, UC-05, and UC-15 are specific instances of this pattern. | +| **Postconditions** | Downstream tasks can access the propagated processing state they need. | + +--- + +### UC-09 — Provide a Transient Intra-Stage Workspace + +| | | +|-------|---------| +| **Actor(s)** | Aggregate tasks, child tasks, task execution framework | +| **Summary** | Within a stage, the context must be usable as a temporary working space for child-task execution. Child tasks must be able to modify context state destructively while they run, including adding, removing, or replacing tentative calibration and processing state, without requiring explicit cleanup logic. Only outputs that are explicitly accepted into the enclosing task's context should survive stage execution. | +| **Invariant** | State changes made while executing against a temporary child-task context do not escape that workspace unless they are explicitly accepted and merged. | +| **Postcondition** | When a child task finishes, the enclosing task retains only the accepted state changes; unaccepted mutations to the temporary workspace are discarded. | + +--- +### UC-10 — Support Multiple Orchestration Drivers + +| | | +|-------|---------| +| **Actor(s)** | Operations / automated processing (PPR-driven batch), pipeline developer / power user (interactive), recipe executor | +| **Summary** | The context is created and consumed by multiple front-ends: PPR command lists, XML procedures, or interactive task calls. The state stored by the context must remain consistent and usable regardless of which driver created or resumed it. It must be creatable and resumable from non-interactive and interactive drivers, support driver-injected run metadata, and tolerate partial execution controls and breakpoint-driven stop/resume semantics. | +| **Invariant** | Processing state is consistent and usable regardless of which orchestration driver created or resumed it, and success/failure signals are produced when appropriate. | + +--- + +### UC-11 — Save and Restore a Processing Session + +| | | +|-------|---------| +| **Actor(s)** | Pipeline operator, workflow orchestration layer, pipeline developer | +| **Summary** | The context must be able to serialize the complete processing state to disk (all observation data, calibration state, execution history, imaging state, project metadata, etc.) and later restore it so that processing can resume from the saved point. The serialization must preserve enough state to resume; backward compatibility across pipeline releases is not guaranteed. On restore, paths must be adaptable to a new filesystem environment. | +| **Postconditions** | After restore, the processing state is operationally equivalent to the saved state for supported resume workflows, and processing can continue from the specified point. | + +--- + +### UC-12 — Provide State to Parallel Workers + +| | | +|-------|---------| +| **Actor(s)** | Workflow orchestration layer, parallel worker processes | +| **Summary** | When work is distributed across parallel workers, each worker needs read-only access to the current processing state (observation metadata, calibration state). The context must provide a mechanism for workers to obtain a consistent snapshot of that state. Workers must not be able to modify the shared processing state directly. The snapshot mechanism must support efficient distribution to workers. | +| **Invariant:**| Worker processes cannot modify shared processing state directly. | +| **Postconditions** | After distribution, each worker has a consistent, read-only view of the processing state for the duration of its work. | + +--- + +### UC-13 — Aggregate Results from Parallel Workers + +| | | +|-------|---------| +| **Actor(s)** | Workflow orchestration layer | +| **Summary** | After parallel workers complete, the context must collect their individual results and incorporate them into the shared processing state. The aggregation must be safe (no conflicting concurrent writes) and complete before the next sequential step begins. | +| **Postconditions** | The processing state reflects the combined outcomes of all parallel workers. | + +--- + +### UC-14 — Provide Read-Only State for Reporting + +| | | +|-------|---------| +| **Actor(s)** | Report generators (weblog, quality reports, reproducibility scripts, pipeline statistics) | +| **Summary** | The context must provide read-only access to the observation metadata, project metadata, execution history (including per-stage domain-specific outputs such as flag summaries and plot references), QA outcomes, log references, and path information needed to generate reporting products such as weblogs, quality reports, reproducibility scripts, and pipeline statistics. | +| **Postconditions** | Reports accurately reflect the processing state at the time of generation. | + +--- + +### UC-15 — Support QA Evaluation and Store Quality Assessments + +| | | +|-------|---------| +| **Actor(s)** | QA scoring framework, report generators | +| **Summary** | After each processing step completes, the context must make the relevant processing state available to QA handlers so they can evaluate the outcome against quality thresholds, which may depend on e.g. telescope, project parameters, or observation properties. The resulting quality scores must be recorded and remain retrievable for reporting. | +| **Postconditions** | Quality scores are associated with the relevant processing step and accessible to reports. | + +--- + +### UC-16 — Support Inspection and Debugging + +| | | +|-------|---------| +| **Actor(s)** | Pipeline developer, pipeline operator, CI harnesses | +| **Summary** | The context must allow an operator to inspect the current processing state, for example: which datasets are registered, what calibrations exist, how many steps have completed, and what their outcomes were. On failure, a snapshot of the state must be available for post-mortem analysis. | +| **Invariant** | The current processing state is inspectable at any point during execution, and sufficient information is retained to diagnose failures after the fact. | + +--- + +### UC-17 — Manage Telescope- and Array-Specific State + +| | | +|-------|---------| +| **Actor(s)** | Telescope-specific tasks and heuristics, array-specific tasks and heuristics | +| **Summary** | The context must support conditional telescope-specific and array-specific extensions to the processing state. These extensions must be available to the tasks and heuristics that need them, including cases where one array mode within a telescope family has materially different state requirements from another. Generic pipeline components must not depend on or require knowledge of those telescope- or array-specific extensions. | +| **Invariant** | Telescope- and array-specific extensions are present only for runs that require them, available to the tasks that need them, and are never assumed by shared pipeline code. | + +--- + +### UC-18 — Provide State for Product Export + +| | | +|-------|---------| +| **Actor(s)** | Export tasks | +| **Summary** | The context must make the datasets, calibration state, image products, reports, scripts, path information, project identifiers, and any other information needed for export available through the processing state so export tasks can assemble them into a deliverable product package. | +| **Invariant** | The information needed to assemble a deliverable product package is accessible through the processing state. |