Completing Workflow can cause a problem if another workflow is submitted at that time

[pagrubel]

I was able to duplicate this using the cancel function and immediately submit a workflow.
The newly submitted workflow never really runs and the one that should have been archived did not get archived.

Name	ID	Status
copy-container	d648ad	Archived
cgt	28a225	Archived/Cancelled
cgt	0182a0	Archived/Cancelled
cgt	06db80	Archived/Cancelled
cgt	c8cce0	Archived/Cancelled
cgt	12873a	Cancelled
cgt	565489	Running
(beedev-env) pagrubel@darwin-fe1 workdir$ beeflow query 56
Running
task_name    task_state
-----------  ------------
cat          SUBMIT
grep0        WAITING
grep1        WAITING
tar          WAITING
(beedev-env) pagrubel@darwin-fe1 workdir$ beeflow query 56
Running
task_name    task_state
-----------  ------------
cat          SUBMIT
grep0        WAITING
grep1        WAITING
tar          WAITING

Here are the entries from the task manager log:

WFM not responding when sending task updates.
[2025-11-20 15:01:48,198] beeflow.task_manager.background:process_queues(): Traceback (most recent call last):
  File "/vast/home/pagrubel/beedev-env/lib/python3.12/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/vast/home/pagrubel/beedev-env/lib/python3.12/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vast/home/pagrubel/beedev-env/lib/python3.12/site-packages/flask_restful/__init__.py", line 467, in wrapper
    resp = resource(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vast/home/pagrubel/beedev-env/lib/python3.12/site-packages/flask/views.py", line 109, in view
    return current_app.ensure_sync(self.dispatch_request)(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vast/home/pagrubel/beedev-env/lib/python3.12/site-packages/flask_restful/__init__.py", line 582, in dispatch_request
    resp = meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/vast/home/pagrubel/BEE/BEE/beeflow/wf_manager/resources/wf_update.py", line 87, in put
    self.update_task_state(state_update)
  File "/vast/home/pagrubel/BEE/BEE/beeflow/wf_manager/resources/wf_update.py", line 184, in update_task_state
    self.handle_state_change(state_update, task, wfi)
  File "/vast/home/pagrubel/BEE/BEE/beeflow/wf_manager/resources/wf_update.py", line 153, in handle_state_change
    wf_utils.copy_task_output(task, wfi)
  File "/vast/home/pagrubel/BEE/BEE/beeflow/wf_manager/resources/wf_utils.py", line 355, in copy_task_output
    shutil.copy(stdout_path, task_save_path / f"{task.name}-{task.id[:4]}.out")
  File "/projects/opt/centos8/x86_64/miniconda3/py312_24.11.1/lib/python3.12/shutil.py", line 435, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/projects/opt/centos8/x86_64/miniconda3/py312_24.11.1/lib/python3.12/shutil.py", line 262, in copyfile
    with open(dst, 'wb') as fdst:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/vast/home/pagrubel/.beeflow/workflows/c8cce0a2ca2847cfa49c2428eea367ef/grep0-ff32/grep0-ff32.out'

[2025-11-20 15:01:48,198] beeflow.task_manager.background:process_queues(): WFM not responding when sending task updates.
[2025-11-20 15:01:53,202] beeflow.task_manager.background:process_queues(): Traceback (most recent call last):
  File "/vast/home/pagrubel/beedev-env/lib/python3.12/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/vast/home/pagrubel/beedev-env/lib/python3.12/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vast/home/pagrubel/beedev-env/lib/python3.12/site-packages/flask_restful/__init__.py", line 467, in wrapper
    resp = resource(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vast/home/pagrubel/beedev-env/lib/python3.12/site-packages/flask/views.py", line 109, in view
    return current_app.ensure_sync(self.dispatch_request)(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vast/home/pagrubel/beedev-env/lib/python3.12/site-packages/flask_restful/__init__.py", line 582, in dispatch_request
    resp = meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/vast/home/pagrubel/BEE/BEE/beeflow/wf_manager/resources/wf_update.py", line 87, in put
    self.update_task_state(state_update)
  File "/vast/home/pagrubel/BEE/BEE/beeflow/wf_manager/resources/wf_update.py", line 184, in update_task_state
    self.handle_state_change(state_update, task, wfi)
  File "/vast/home/pagrubel/BEE/BEE/beeflow/wf_manager/resources/wf_update.py", line 153, in handle_state_change
    wf_utils.copy_task_output(task, wfi)
  File "/vast/home/pagrubel/BEE/BEE/beeflow/wf_manager/resources/wf_utils.py", line 355, in copy_task_output
    shutil.copy(stdout_path, task_save_path / f"{task.name}-{task.id[:4]}.out")
  File "/projects/opt/centos8/x86_64/miniconda3/py312_24.11.1/lib/python3.12/shutil.py", line 435, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/projects/opt/centos8/x86_64/miniconda3/py312_24.11.1/lib/python3.12/shutil.py", line 262, in copyfile
    with open(dst, 'wb') as fdst:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/vast/home/pagrubel/.beeflow/workflows/c8cce0a2ca2847cfa49c2428eea367ef/grep0-ff32/grep0-ff32.out'

[2025-11-20 15:01:53,203] beeflow.task_manager.background:process_queues(): WFM not responding when sending task updates.
[2025-11-20 15:01:58,210] beeflow.task_manager.background:process_queues(): Traceback (most recent call last):
  File "/vast/home/pagrubel/beedev-env/lib/python3.12/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/vast/home/pagrubel/beedev-env/lib/python3.12/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vast/home/pagrubel/beedev-env/lib/python3.12/site-packages/flask_restful/__init__.py", line 467, in wrapper
    resp = resource(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vast/home/pagrubel/beedev-env/lib/python3.12/site-packages/flask/views.py", line 109, in view
    return current_app.ensure_sync(self.dispatch_request)(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vast/home/pagrubel/beedev-env/lib/python3.12/site-packages/flask_restful/__init__.py", line 582, in dispatch_request
    resp = meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/vast/home/pagrubel/BEE/BEE/beeflow/wf_manager/resources/wf_update.py", line 87, in put
    self.update_task_state(state_update)
  File "/vast/home/pagrubel/BEE/BEE/beeflow/wf_manager/resources/wf_update.py", line 184, in update_task_state
    self.handle_state_change(state_update, task, wfi)
  File "/vast/home/pagrubel/BEE/BEE/beeflow/wf_manager/resources/wf_update.py", line 153, in handle_state_change
    wf_utils.copy_task_output(task, wfi)
  File "/vast/home/pagrubel/BEE/BEE/beeflow/wf_manager/resources/wf_utils.py", line 355, in copy_task_output
    shutil.copy(stdout_path, task_save_path / f"{task.name}-{task.id[:4]}.out")
  File "/projects/opt/centos8/x86_64/miniconda3/py312_24.11.1/lib/python3.12/shutil.py", line 435, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/projects/opt/centos8/x86_64/miniconda3/py312_24.11.1/lib/python3.12/shutil.py", line 262, in copyfile
    with open(dst, 'wb') as fdst:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/vast/home/pagrubel/.beeflow/workflows/c8cce0a2ca2847cfa49c2428eea367ef/grep0-ff32/grep0-ff32.out'

[2025-11-20 15:01:58,211] beeflow.task_manager.background:process_queues(): WFM not responding when sending task updates.

[pagrubel]

Would upon Cancelling a workflow would cancelling any running jobs and halting any from being submitted before Cancelling the workflow help?
Also, would a quick fix be to not allow any submit commands until Cancelling workflows are archived?