Workflow continues to run after cancelled on dashboard #1147
Description
Description:
After cancelling a workflow on dashboard, There is a warning 'Thread 6330920960 with run id 3be4c09f-7ca1-482d-aeee-30fd50f9eb1c is still running after cancellation'. The code still continue to run until it finishes.
Logs:
[DEBUG] 🪓 -- 2024-12-23 15:34:48,817 - sending heartbeat
[INFO] 🪓 -- 2024-12-23 15:34:51,904 - rx: start step run: 3be4c09f-7ca1-482d-aeee-30fd50f9eb1c/first-python-workflow:step1
[DEBUG] 🪓 -- 2024-12-23 15:34:51,905 - tx: event: first-python-workflow:step1/1
[INFO] 🪓 -- 2024-12-23 15:34:51,905 - run: start step: first-python-workflow:step1/3be4c09f-7ca1-482d-aeee-30fd50f9eb1c
[DEBUG] 🪓 -- 2024-12-23 15:34:51,906 - tx: event: first-python-workflow:step1/1
INFO:root:executed step1
[DEBUG] 🪓 -- 2024-12-23 15:34:51,906 - start time: 0.0013298988342285156
0
[DEBUG] 🪓 -- 2024-12-23 15:34:52,826 - sending heartbeat
1
2
3
4
[DEBUG] 🪓 -- 2024-12-23 15:34:56,871 - sending heartbeat
5
[INFO] 🪓 -- 2024-12-23 15:34:57,469 - rx: cancel step run: 3be4c09f-7ca1-482d-aeee-30fd50f9eb1c
[INFO] 🪓 -- 2024-12-23 15:34:57,469 - cancel: step run: /3be4c09f-7ca1-482d-aeee-30fd50f9eb1c
[DEBUG] 🪓 -- 2024-12-23 15:34:57,470 - cancelling step...
6
[WARNING] 🪓 -- 2024-12-23 15:34:58,471 - Thread 6330920960 with run id 3be4c09f-7ca1-482d-aeee-30fd50f9eb1c is still running after cancellation. This could cause the thread pool to get blocked and prevent new tasks from running.
7
8
[DEBUG] 🪓 -- 2024-12-23 15:35:00,933 - sending heartbeat
9
10
11
12
[DEBUG] 🪓 -- 2024-12-23 15:35:04,938 - sending heartbeat
13
14
15
16
[DEBUG] 🪓 -- 2024-12-23 15:35:08,940 - sending heartbeat
17
18
19
[DEBUG] 🪓 -- 2024-12-23 15:35:12,944 - sending heartbeat
[DEBUG] 🪓 -- 2024-12-23 15:35:16,949 - sending heartbeat
[DEBUG] 🪓 -- 2024-12-23 15:35:21,017 - sending heartbeat
Expected behaviour:
It should stop logging new numbers after the workflow is cancelled.
Worker code for reproduction:
from hatchet_sdk import Context, Hatchet, ClientConfig
from dotenv import load_dotenv
import logging
logging.basicConfig(level=logging.INFO)
LOG = logging.getLogger()
load_dotenv()
hatchet = Hatchet(
debug=True,
config=ClientConfig(
logger=LOG,
),
)
@hatchet.workflow(name="first-python-workflow")
class MyWorkflow:
@hatchet.step(retries=3)
def step1(self, context: Context):
LOG.info("executed step1")
import time
i = 0
while i < 20:
print(i)
i += 1
time.sleep(1)
return {
"result": "success"
}
if __name__ == "__main__":
worker = hatchet.worker('first-worker')
worker.register_workflow(MyWorkflow())
worker.start()
Configuration Details:
- Self hosted with https://docs.hatchet.run/self-hosting/docker-compose
- Hatched SDK version 0.42.5
- Python version: 3.11.9
- macOS version: 15.1.1