Skip to content

Document usage of NODENAME for mount_utils.py #49

@hacktobeer

Description

@hacktobeer

When a worker wants to mount a disk we need to make (and document) that NODENAME is set in the docker-compose configuration across all (mounting) workers running on the same host. This to ensure that the redis lock key is the same across all containers running on the same host.

If NODENAME is NOT set and a worker tries to mount it can create race condititions that are hard to debug and can generate errors like below:

[2025-05-30 07:19:17,283: INFO/ForkPoolWorker-4] Redis lock succesfully set: cf3f5472d882-/dev/nbd0 for cf3f5472d882-/dev/nbd0
[2025-05-30 07:19:17,457: INFO/ForkPoolWorker-4] qemu-nbd: success creating /dev/nbd0 for /usr/share/openrelik/data/artifacts/4246e4aee76a474db828fa6e934b94e5/f6f1a4386a7d4b28a1f60c0398b373cd.qcow2
[2025-05-30 07:19:17,691: ERROR/ForkPoolWorker-4] fdisk: failed probing /dev/nbd0 for /usr/share/openrelik/data/artifacts/4246e4aee76a474db828fa6e934b94e5/f6f1a4386a7d4b28a1f60c0398b373cd.qcow2: fdisk: cannot open /dev/nbd0: Invalid argument

[2025-05-30 07:19:17,691: INFO/ForkPoolWorker-4] Error: Error fdisk: failed probing: fdisk: cannot open /dev/nbd0: Invalid argument

In above case it looked like a succesful redis lock was acquired for /dev/nbd0 with key cf3f5472d882-/dev/nbd0. In this case NODENAME is not set and the first part of the key is the hostname IN the container. As other containers also try to mount and have a different hostname this will succeed while in reality the device is already in use by another worker. The first part of the key needs to be the same for workers running on the same server/node.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions