From 2c27313ad2decfc1535e5e85f2121098ea1e57a6 Mon Sep 17 00:00:00 2001 From: SJaffa Date: Fri, 23 May 2025 16:25:01 +0100 Subject: [PATCH 1/4] Document workflow output --- .../getting_started/running_workflows.rst | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/docs/source/user/getting_started/running_workflows.rst b/docs/source/user/getting_started/running_workflows.rst index ceb7362..d62081c 100644 --- a/docs/source/user/getting_started/running_workflows.rst +++ b/docs/source/user/getting_started/running_workflows.rst @@ -66,3 +66,29 @@ Sometimes you might want to cancel a workflow that is running. Use where ``WORKFLOW_REF`` is either the path to the workflow directory, or the ID of the workflow displayed by ``{{ app_module }} show``. + +Workflow output +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In your working directory, a new directory will be created with the name of the workflow and the date and time of submission. +This directory contains the output of the workflow, including your results files, the log files and any other files generated by the workflow. + +The directory structure is as follows: +``` +workflow_name/ +├── artifacts/ +│   ├── workflow.yaml +│   ├── submissions/ +│   └── tasks/ +├── execute/ +├── metadata/ +└── parameters/ +``` + +The ``artifacts`` directory contains a copy of the workflow file that you ran, the submission files, and the task files. +The ``submissions`` directory contains the jobscrits which are submitted to the scheduler and any log and error files generated by the software as it runs. +If your workflow crashes, you can check the ``stdout/stderr`` files in this directory to see what happended. +The ``tasks`` directory contains a folder for each task specified in the workflow, and the output files you save from each task will be saved in this directory. + +The ``execute``, ``metadata``, and ``parameters`` directories contain binary files and log files used by {{ app_name }}. +You will not normally need to look at these files, but they are available there for advanced debugging and checkpointing. \ No newline at end of file From e74afdec280285b72fd4b4a7f20c902dd5e234d1 Mon Sep 17 00:00:00 2001 From: SJaffa Date: Fri, 23 May 2025 16:31:48 +0100 Subject: [PATCH 2/4] typos --- docs/source/user/getting_started/running_workflows.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/user/getting_started/running_workflows.rst b/docs/source/user/getting_started/running_workflows.rst index d62081c..0742fbe 100644 --- a/docs/source/user/getting_started/running_workflows.rst +++ b/docs/source/user/getting_started/running_workflows.rst @@ -75,7 +75,7 @@ This directory contains the output of the workflow, including your results files The directory structure is as follows: ``` -workflow_name/ +workflow_nam_YYYY-MM-DD_timestamp/ ├── artifacts/ │   ├── workflow.yaml │   ├── submissions/ @@ -86,7 +86,7 @@ workflow_name/ ``` The ``artifacts`` directory contains a copy of the workflow file that you ran, the submission files, and the task files. -The ``submissions`` directory contains the jobscrits which are submitted to the scheduler and any log and error files generated by the software as it runs. +The ``submissions`` directory contains the jobscripts which are submitted to the scheduler and any log and error files generated by the software you ran. If your workflow crashes, you can check the ``stdout/stderr`` files in this directory to see what happended. The ``tasks`` directory contains a folder for each task specified in the workflow, and the output files you save from each task will be saved in this directory. From f9de70ebf075ab028088346c328454b98875141b Mon Sep 17 00:00:00 2001 From: SJaffa Date: Fri, 23 May 2025 16:32:24 +0100 Subject: [PATCH 3/4] typo --- docs/source/user/getting_started/running_workflows.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user/getting_started/running_workflows.rst b/docs/source/user/getting_started/running_workflows.rst index 0742fbe..a739bd3 100644 --- a/docs/source/user/getting_started/running_workflows.rst +++ b/docs/source/user/getting_started/running_workflows.rst @@ -75,7 +75,7 @@ This directory contains the output of the workflow, including your results files The directory structure is as follows: ``` -workflow_nam_YYYY-MM-DD_timestamp/ +workflow_name_YYYY-MM-DD_timestamp/ ├── artifacts/ │   ├── workflow.yaml │   ├── submissions/ From f227042fee8ec1b52d0a1ad36b97dc49b68ba933 Mon Sep 17 00:00:00 2001 From: S Jaffa Date: Thu, 29 May 2025 10:10:33 +0100 Subject: [PATCH 4/4] Review suggestions Co-authored-by: Adam Plowman --- .../source/user/getting_started/running_workflows.rst | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/docs/source/user/getting_started/running_workflows.rst b/docs/source/user/getting_started/running_workflows.rst index a739bd3..22fce0d 100644 --- a/docs/source/user/getting_started/running_workflows.rst +++ b/docs/source/user/getting_started/running_workflows.rst @@ -85,10 +85,9 @@ workflow_name_YYYY-MM-DD_timestamp/ └── parameters/ ``` -The ``artifacts`` directory contains a copy of the workflow file that you ran, the submission files, and the task files. -The ``submissions`` directory contains the jobscripts which are submitted to the scheduler and any log and error files generated by the software you ran. -If your workflow crashes, you can check the ``stdout/stderr`` files in this directory to see what happended. -The ``tasks`` directory contains a folder for each task specified in the workflow, and the output files you save from each task will be saved in this directory. +The ``artifacts`` directory contains a copy of the workflow template file that you submitted, the submission files, and the task artifact files. +The ``submissions`` directory contains generated jobscripts that are submitted to the scheduler, standard output and error stream files from those jobscripts, and app log files (if {{ app_name }} is configured to generate them). +If your workflow crashes, you should first check the files in the ``js_std`` directory to see what happened. +The ``tasks`` directory contains files associated with task execution that we wish to keep (for example input and output files for a simulation). We can control which files are kept within the task schema. -The ``execute``, ``metadata``, and ``parameters`` directories contain binary files and log files used by {{ app_name }}. -You will not normally need to look at these files, but they are available there for advanced debugging and checkpointing. \ No newline at end of file +The ``execute`` directory contains the working directories for the tasks, and so any output files generated by the software that is invoked will be here. The ``metadata`` and ``parameters`` directories contain binary files, corresponding to workflow structural metadata, and input and output parameters, respectively. You will not normally need to look at these files, but they are available there for advanced debugging and checkpointing. \ No newline at end of file