Add delayed jobtemp publishing with spill plane support for Thor

[Copilot]

Adds @delayJobTempPublish configuration option to write Thor jobtemps to the spill plane and defer Dali publishing until workunit transitions, reducing I/O to persistent storage.

Changes

Plane Selection (common/thorhelper/thorwrite.cpp)

• Modified getDefaultWritePlane() to route jobtemps to spill plane when @delayJobTempPublish is enabled

Stowing Mechanism (thorlcr/mfilemanager/thmfilemanager.cpp)

• Added StowedJobTemp structure tracking logical name, WUID, and file reference
• Implemented in-memory stowing array with workunit-scoped lifecycle
• Modified publish() to stow jobtemps instead of immediate Dali attachment when feature enabled

Workunit Transition Handling

• Detects WUID changes in both lookup() and publish() paths
• Auto-publishes stowed jobtemps from previous workunit on transition
• Lazy initialization of currentWuid on first operation

Lookup Path

• Enhanced lookup() to check stowed set before Dali query
• Removes consumed jobtemps from stowed set on retrieval
• Falls through to existing timeout/retry logic if not found

Configuration

<Thor @delayJobTempPublish="true">

When enabled, jobtemps written to spill plane remain unpublished until:

1. Next job references different workunit (triggers publish of previous WU's jobtemps)
2. Same-workunit jobs consume directly from stowed set without Dali roundtrip

Known Limitations

• Linear search through stowed jobtemps (O(n) lookup)
• Duplicate storage in both fileMap cache and stowing array
• Stowed jobtemps cleared on shutdown without publishing (no workunit context available)

Original prompt

Task Summary: Enhance Thor’s handling of jobtemps by introducing a new option to conditionally utilize the spill plane while ensuring jobtemps are stowed and preserved between jobs.

Task Description:Currently, in Thor jobtemps are typically written to the data plane as determined by getDefaultJobTempPlane. Modify this behavior by adding a configuration option (delayJobTempPublish) that, when enabled, directs Thor to use the spill plane instead of the data plane for jobtemps. This must account for specific behaviors:

1. Stowing and Retention of Jobtemps:

• Delayed unpublished logical jobtemps written to the spill plane must not be discarded after the job completes.

• Instead, they should be stowed and preserved for subsequent use if needed.

2. Handling Subsequent Jobs:

• When a new job is received, if it corresponds to the same workunit as the previous job:

• Look up jobtemps in the stowed set.

• If a match is found:

• Use the jobtemp.

• Remove it from the stowed set once consumed.

• If no match is found in the stowed set, check if the jobtemp has been published. If so, use the published version.

• If neither option is available, wait on a workunit-specific subscription (e.g., via SDS notifications) and periodically re-check until the jobtemp is ready.

3. Transitioning Between Workunits:

• If the next job received is not associated with the previous workunit, Thor should:

• Publish all stowed jobtemps associated with the previous workunit before starting the new job.

• This ensures no data is lost and smooth transitions between workunits.

Notes: Use the existing unpublished logical spill file mechanisms as a reference for tracking and managing stowed jobtemp files. Implement the changes in a way that integrates seamlessly with Thor’s current file manager and job dependency management.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.