Check Point Restart Discussion

[pagrubel]

Towards Milestone due end of FY22:

Tim:
The cwl could have a hint if the step has check point/ restart need

TaskManager should know the exit code from an incomplete task, slurm will tell you

• if the job is killed for wallclock limit you'll know
• if the job crashes, we need to talk to a code team person probably
  ...when does a user know that a job should be restarted?
  • in the case of an application crash probably don't restart
  • in the case of a node crash (the application was fine) you probably do restart

[pagrubel]

If a step has check point/ restart hint and we are going to re run it we can have a new state for that.
The task manager can let the WFM know by having new state that goes into the database indicating the step needs to be restarted. TM should also verify that the checkpoint file exists and communicate that to the WFM. Since a new job will have to be started when that happens the TM can update the meta data to (WFM, GDB) with the new job, preserving any previous job info in the metadata.