Document current Pipeline Context use cases by sbooth-nrao · Pull Request #17 · casangi/radps-context

sbooth-nrao · 2026-03-11T23:46:24Z

Adds document that catalogues how the current ALMA/VLA pipeline uses its Context object.

context_use_cases_legacy_pipeline.md — 17 use cases (UC-01 – UC-17) describing what the current pipeline context does. The initial draft was merged from drafts by Berry and Booth.
docs/context_current_pipeline_appendix.md - an appendix which describes the implementation of the current context use cases.

krlberry

@sbooth-nrao Thank you for your work on combining our drafts into this document. I’ve left several comments for discussion. I have a couple of overall comments:

There is a lot of non-use case content in this document. My suggestion is that we either trim this for brevity or move it to an appendix so the use cases are readable on their own.
Several current use cases reference current implementation details: specific task names, class names, access patterns — at a level of specificity that makes it hard to evaluate whether RADPS needs to satisfy them. I think these should be abstracted up: written in terms of what the system needs to do, not how the current pipeline does it. These implementation details could be moved to an appendix for reference.
For the future use cases, I think we should note whether each traces to a specific RADPS requirement, an aspect of the RADPS design that implies this use case, or is more of a ‘wish list’ item that may belong in a separate doc or section.

docs/context_use_cases_current_pipeline.md

krlberry · 2026-03-16T05:30:09Z

docs/context_use_cases_current_pipeline.md

+
+---
+
+## 6. Architectural Observations


Sections 6-8 are interesting and useful for discussing the future context design, but I don't think that this document is the best place for them. I'd prefer to keep it streamlined and focused so when we pass this document around for review and feedback about missing use cases the most relevant content is clear and isolated.

krlberry · 2026-03-16T05:47:06Z

docs/context_use_cases_current_pipeline.md

+- Role-based access to context fields
+- Audit logging of all context mutations
+
+### FUC-07 — Partial Re-Execution / Targeted Stage Re-Run


I think this is a great idea, and I am also wondering if it is in scope? Can this be tied to a requirement for RADPS?

docs/context_use_cases_current_pipeline.md

krlberry · 2026-03-16T05:48:20Z

docs/context_use_cases_current_pipeline.md

+- A query API (REST, gRPC, or GraphQL)
+- Type definitions shared across languages
+
+### FUC-04 — Streaming / Incremental Processing


Can this be tied to a requirement from RADPS?

krlberry · 2026-03-16T05:48:35Z

docs/context_use_cases_current_pipeline.md

+- Artifact references rather than filesystem paths for cal tables and images
+- Tasks that can operate on remote datasets without requiring local copies
+
+### FUC-03 — Multi-Language / Multi-Framework Access to Context


Nice to have -- is this a requirement from RADPS?

krlberry · 2026-03-16T05:49:02Z

docs/context_use_cases_current_pipeline.md

+- A merge/reconciliation step when concurrent results are accepted
+- Explicit declaration of which context fields each task reads and writes
+
+### FUC-02 — Cloud / Distributed Execution Without Shared Filesystem


I think we could probably tie non-local execution to RADPS requirements.

If the plan is to map these "GAP" use cases to RADPS requirements, should they even be in this document?

This is a very good point. After reflection, I think my questions about specifically "RADPS requirements" were too narrow.

Here is my updated thought on this: I think each future use case should identify its source — whether that's an explicit RADPS requirement, something implied by the RADPS architecture, a known pain point, or something else. Without that it's hard to evaluate whether they belong here.

docs/context_use_cases_current_pipeline.md

…ndix file; updated some wording choices for more accurate language and removed deployment-level GAP scenario

krlberry

I left some more comments for discussion.

docs/context_use_cases_current_pipeline.md

krlberry · 2026-03-16T16:44:49Z

docs/context_use_cases_current_pipeline.md

+- A merge/reconciliation step when concurrent results are accepted
+- Explicit declaration of which context fields each task reads and writes
+
+### FUC-02 — Cloud / Distributed Execution Without Shared Filesystem


This is a very good point. After reflection, I think my questions about specifically "RADPS requirements" were too narrow.

Here is my updated thought on this: I think each future use case should identify its source — whether that's an explicit RADPS requirement, something implied by the RADPS architecture, a known pain point, or something else. Without that it's hard to evaluate whether they belong here.

docs/context_use_cases_current_pipeline.md

… document

…dix.

…agraphs

…secases.

…ding

…ases.

…t UC-06 into two use cases and update use case numbering

…n UC-04.

…date use case titles and numbering.

…n the pipeline (the Context with backticks).

…ecision making in downstream tasks. Make other assorted wording updates including removing references to removed use cases and standardizing actor names.

…llibrary

…o blarn

Use case edit suggestions

indebetouw · 2026-03-31T13:57:15Z

UC-1 metadata: also need cross-MS matching and lookup which I'd make a separate item
current mechanisms include
-virtual spw name<>ID translation, which needs expansion to different kinds of match - exact, for some calibration tasks, and partial, for overlapping windows that can be imaged together.
-data type tracking across MS and MS column

This is one place where reinventing the wheel could be beneficial, because the current implementations use a "single master MS" and all MS were originally assumed to have the exact same sources, spw IDs, etc.

krlberry · 2026-03-31T22:25:46Z

Here's a PDF version of this: context_use_cases_current_pipeline.pdf

dmuders · 2026-04-07T14:56:28Z

docs/context_current_pipeline_appendix.md

+**Implementation notes** — the current pipeline satisfies these needs through two different propagation paths:
+
+1. **Immediate state propagation** — `Results.merge_with_context(context)` updates calibration library, image libraries, and more so later tasks can access the current processing state directly.
+2. **Serialized Results** — tasks read `context.results` to find outputs from earlier stages when those outputs are needed from the recorded results rather than from merged shared state. For example:


This pattern has crept in over time. The original idea was to not be dependent on results object parsing outside their native tasks. Also using explicitly the "previous" result makes assumptions about the recipe sequence. But even checking for an explicit results object type then still needs knowledge about the class structure of another task. That's why we tried using the extra attributes like "clean_list_pending" etc. Though they are, as you wrote, a bit ad-hoc and should probably at least have had a container class.

Thanks Dirk, I will add some clarifying language regarding the intended behavior versus adapted behavior, including specific example in the code where each is used. We can also make sure to include this as an example of context creep, where an intended behavior gets lost without strict contractual definitions.

dmuders · 2026-04-07T14:58:50Z

docs/context_current_pipeline_appendix.md

+
+**Implementation notes** — `WebLogGenerator.render(context)` in `pipeline/infrastructure/renderer/htmlrenderer.py`:
+
+- Reads `context.results` — unpickled from `ResultsProxy` objects, iterated for every renderer


Is there really a case where all results are automatically unpickled? I thought one always had to call "read()".

You are correct Dirk. There is a line in the htmlrenderer.py:1897 that does a mass read of all the result proxies that was mistaken as automatic unpickling of all the result objects. I will modify the language to be more representative of the actual behavior.

dmuders · 2026-04-07T15:01:08Z

docs/context_current_pipeline_appendix.md

+  - Most handlers call `context.observing_run.get_ms(vis)` to look up metadata for scoring (antenna count, channel count, SPW properties, field intents)
+  - Some handlers check `context.imaging_mode` to branch on VLASS-specific scoring
+  - Others check things in `context.observing_run`, `context.project_structure`, or the callibrary (`context.callibrary`)
+- Scores are appended to `result.qa.pool`, so the scores are stored on the results rather than directly on the context. 


I don't remember if this was due to some size consideration too. We can potentially have many QA score objects in the pool if there is one per detailed data selection (field/spw/pol/ant/baseline/...).

Added language to the UC-15 implementation notes indicating current implementation behavior. Also made a note to create explicit rules to restrict this in future designs.

dmuders · 2026-04-07T15:04:18Z

docs/context_gap_use_cases.md

+- GAP-03: Provenance and reproducibility — requires immutable per-attempt records, input hashing, and lineage capture.
+- GAP-04: Partial re-execution / targeted rerun — requires explicit dependency tracking and invalidation semantics at the context level.
+- GAP-05: External system integration — requires stable identifiers, event subscriptions/webhooks, and exportable summaries/manifests.
+- GAP-06: Multi-language access — requires a language-neutral schema and API for context state and artifact queries.


Do you mean "programming language"?

For a low barrier approach it would be good to have something like a middleware so that the local language API does not need to expose the actual structure of how the context is stored. And since we/I think that the dev team should be able to add new items quickly, some "standard" data types (incl. dictionaries or equivalent) should be readily available.

I renamed GAP-06 to explicitly mention Programming Language and included a recommendation for a stable middleware layer.

docs/context_current_pipeline_appendix.md

…-12/UC-14 impl notes; add GAP-08 (cross-MS matching); update GAP-06 title and summary

tnakazato

@sbooth-nrao @krlberry thank you very much for your work. The document is comprehensive and is very good high-level summary of the usecase. I made a few comments. I would appreciate it if you could take a look.

docs/context_use_cases_current_pipeline.md

…dd GAP-08, refine GAP-06)

commit for combined current context use cases draft

24395dc

krlberry reviewed Mar 16, 2026

View reviewed changes

sbooth-nrao commented Mar 16, 2026

View reviewed changes

docs/context_use_cases_current_pipeline.md Outdated Show resolved Hide resolved

moved analysis/commentary/design recommendations into a separate appe…

ae2a245

…ndix file; updated some wording choices for more accurate language and removed deployment-level GAP scenario

krlberry reviewed Mar 18, 2026

View reviewed changes

sbooth-nrao commented Mar 18, 2026

View reviewed changes

docs/context_use_cases_current_pipeline.md Outdated Show resolved Hide resolved

sbooth-nrao and others added 19 commits March 18, 2026 21:48

integrated review notes for the draft of the current pipeline context…

f5125ae

… document

Clean up GAP use cases and move exploratory future use cases to appen…

51e7f38

…dix.

Add descriptions of all use case fields. Update wording for intro par…

f958493

…agraphs

Update wording and change invariant vs postcondition label for some u…

4c27b9a

…secases.

Move external systems integration use case back to GAP and update wor…

f00edab

…ding

Update wording for several use cases and merge minor standalone use c…

a753595

…ases.

Update UC-08 based on feedback. Update UC-01 to include updates. Spli…

381da10

…t UC-06 into two use cases and update use case numbering

Add mutability to UC-01; add updating to UC-03; update some wording i…

4a31b20

…n UC-04.

Remove table header labels.

32653f2

Remove future architectural suggestions from appendix document and up…

abb603e

…date use case titles and numbering.

updated wording throughout to be more consistent with scope of document

78f349e

removed GAP and future use cases from docs

5f8cd1d

Update references to the context to not refer to the specific class i…

46be1dd

…n the pipeline (the Context with backticks).

Update qa score use case to refelct that qa scores are not used for d…

506047a

…ecision making in downstream tasks. Make other assorted wording updates including removing references to removed use cases and standardizing actor names.

Fixed typo and removed task category without interactions with the ca…

4b79e0e

…llibrary

updated wording and syntax and removed implementation-specific language

2737431

Update wording in the appendix for clarity and correctness

7323f20

Merge branch 'use_case_edits' of github.com:casangi/radps-context int…

380315a

…o blarn

Merge pull request #18 from casangi/use_case_edits

798473a

Use case edit suggestions

created doc for gap scenarios

8ec2dad

dmuders reviewed Apr 7, 2026

View reviewed changes

sbooth-nrao commented Apr 7, 2026

View reviewed changes

Add UC-09 (intra-stage workspace), renumber UC-09–UC-18; fix UC-04/UC…

c7126e7

…-12/UC-14 impl notes; add GAP-08 (cross-MS matching); update GAP-06 title and summary

tnakazato reviewed Apr 8, 2026

View reviewed changes

docs/context_use_cases_current_pipeline.md Outdated Show resolved Hide resolved

docs/context_use_cases_current_pipeline.md Outdated Show resolved Hide resolved

add UC-09, renumber use-cases; update appendix and GAPs (fix UC-04, a…

3b6e9f4

…dd GAP-08, refine GAP-06)


		Implementation notes — `WebLogGenerator.render(context)` in `pipeline/infrastructure/renderer/htmlrenderer.py`:

		- Reads `context.results` — unpickled from `ResultsProxy` objects, iterated for every renderer

Conversation

sbooth-nrao commented Mar 11, 2026 • edited by krlberry Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krlberry left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

krlberry left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

indebetouw commented Mar 31, 2026

Uh oh!

krlberry commented Mar 31, 2026

Uh oh!

dmuders Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmuders Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tnakazato left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

sbooth-nrao commented Mar 11, 2026 •

edited by krlberry

Loading

krlberry left a comment •

edited

Loading

dmuders Apr 7, 2026 •

edited

Loading

dmuders Apr 7, 2026 •

edited

Loading