Implement cross-cutting Docker integration

[conradbzura]

Description

Add first-class Docker support to wool spanning two components: a container-based worker factory and a Docker discovery backend. Both conform to existing protocols (WorkerLike for the factory, DiscoveryLike for discovery) and compose with the standard WorkerPool API.

DockerWorker — container-based worker factory

A DockerWorker that implements the WorkerLike protocol and is usable anywhere LocalWorker is — including as the worker argument to WorkerPool. On start(), it creates a Docker container running a wool gRPC server image, attaches it to the specified network, labels it with worker metadata, waits for it to become healthy, and populates WorkerMetadata with the container's network-reachable gRPC address. On stop(), it stops and removes the container.

DockerWorker works in both standalone Docker and Swarm environments. In Swarm, the container is attached to an attachable overlay network, making it reachable from any Swarm node — no Swarm service management required.

from wool.runtime.worker.docker import DockerWorker

async with WorkerPool(
    size=8,
    worker=lambda *tags, **kw: DockerWorker(*tags, image="wool-worker:0.3.0", network="wool", **kw),
    discovery=DockerDiscovery(network="wool"),
):
    await some_routine()

DockerDiscovery — unified discovery for standalone, Compose, and Swarm

A DockerDiscovery backend that discovers wool workers running as containers on the Docker Engine, regardless of how they were scheduled (standalone docker run, Compose, or Swarm). Discovery is label-based: the subscriber watches the Engine API event stream for container lifecycle events (start, stop, die) filtered by a configurable set of labels, inspects each matching container, maintains a container cache, and emits worker-added, worker-dropped, and worker-updated events.

The constructor accepts two optional scoping parameters:

• network — scopes discovery to containers on a specific Docker network and resolves each container's IP on that network. When omitted, all matching containers are discovered regardless of network membership.
• labels — a label filter dict passed to the Docker API's event and container list endpoints. Defaults to {"wool.worker": "true"} so it works out of the box with DockerWorker. Override for custom labeling conventions, including Swarm's built-in service labels.

This single type covers standalone Docker, Compose, and Swarm because Swarm tasks are just containers — discovery observes running containers by label and does not need to know what scheduled them. Swarm automatically labels every task container with com.docker.swarm.service.name, so discovering by Swarm service name requires no special backend — just a label filter override.

DockerDiscovery supports two modes:

• Full mode (implements DiscoveryLike) — includes a publisher that labels the current container with worker metadata via the Engine API. Compatible with all pool modes (ephemeral, external, hybrid).
• Subscriber-only mode (implements DiscoverySubscriberLike) — observe-only, no publisher. For environments where an external orchestrator (Swarm, Compose) manages container lifecycles and the pool only needs to discover existing workers (external mode). See Add WorkerPool constructor overload that accepts DiscoverySubscriberLike to support external-only discovery patterns #106 for the pool-level changes needed to accept subscriber-only discovery.

Standalone / Compose usage

from wool.runtime.discovery.docker import DockerDiscovery
from wool.runtime.worker.docker import DockerWorker

# Ephemeral: spawn containers via DockerWorker, discover via default wool label
async with WorkerPool(
    size=8,
    worker=lambda *tags, **kw: DockerWorker(*tags, image="wool-worker:0.3.0", network="wool", **kw),
    discovery=DockerDiscovery(network="wool"),
):
    await some_routine()

# External: discover containers managed by Compose
async with WorkerPool(discovery=DockerDiscovery(network="wool").subscriber):
    await some_routine()

Docker Swarm usage

from wool.runtime.discovery.docker import DockerDiscovery
from wool.runtime.worker.docker import DockerWorker

# External: discover all tasks of a Swarm service by its service name label
async with WorkerPool(
    discovery=DockerDiscovery(
        network="wool-overlay",
        labels={"com.docker.swarm.service.name": "wool-worker"},
    ).subscriber,
):
    await some_routine()

# Hybrid: discover Swarm-managed tasks AND spawn ephemeral burst containers
async with WorkerPool(
    size=4,
    worker=lambda *tags, **kw: DockerWorker(*tags, image="wool-worker:0.3.0", network="wool-overlay", **kw),
    discovery=DockerDiscovery(
        network="wool-overlay",
        labels={"wool.worker": "true"},
    ),
):
    await some_routine()

In the hybrid Swarm example, DockerWorker creates standalone containers on the local daemon with wool.worker=true labels and attaches them to the attachable overlay network. The Swarm service's task containers would also carry wool.worker=true (set in the service's container spec). DockerDiscovery discovers both sets of containers through the same label filter. Swarm handles rescheduling of its own tasks; DockerWorker handles lifecycle of the ephemeral burst containers.

Motivation

LocalWorker spawns workers as subprocesses on the same host, sharing the host's Python environment, memory, and file system. The existing discovery backends cover single-machine (LocalDiscovery via shared memory) and LAN (LanDiscovery via mDNS) deployments. Neither works in containerized environments: shared memory is not available across containers, and mDNS multicast is typically blocked on Docker bridge and overlay networks.

Container-based workers provide process isolation, dependency pinning (each worker can use a different Python environment or OS-level library set), and resource limits (CPU, memory) enforced by the container runtime. A Docker-native discovery backend lets users run wool pools across containers orchestrated by Docker Compose or Docker Swarm without requiring an external service registry. Because Swarm tasks are just containers, a single label-based discovery type with a configurable label filter covers standalone Docker through multi-host Swarm without separate backends. In Swarm environments, standalone containers can join attachable overlay networks to participate alongside Swarm-managed tasks, enabling mixed pools with Swarm services providing an external baseline and DockerWorker containers providing ephemeral burst capacity.

Expected outcome

• DockerWorker in wool/src/wool/runtime/worker/docker.py implements WorkerLike. start() creates a container on the local daemon, attaches it to the specified network, labels it with worker metadata, waits for health, and returns metadata. stop() stops and removes the container. Works on both standalone Docker and Swarm (via attachable overlay networks).
• DockerDiscovery in wool/src/wool/runtime/discovery/docker.py implements DiscoveryLike. The subscriber streams container events filtered by label and optionally by network. The default label filter is {"wool.worker": "true"}; users can override it with custom labels (e.g. {"com.docker.swarm.service.name": "wool-worker"}) to discover Swarm service tasks or any other labeling convention. The publisher labels the current container with worker metadata. Also usable in subscriber-only mode for external pools that discover externally managed containers (requires Add WorkerPool constructor overload that accepts DiscoverySubscriberLike to support external-only discovery patterns #106).
• The Docker SDK (aiodocker or docker) is an optional dependency — importing any Docker component when it is not installed raises a clear error rather than crashing at module load.

Related issues

• Replace durable pool terminology with external throughout the codebase and documentation #105 — Replace "durable" pool terminology with "external"
• Add WorkerPool constructor overload that accepts DiscoverySubscriberLike to support external-only discovery patterns #106 — Accept DiscoverySubscriberLike on WorkerPool for external-only discovery