Docker Restoration and Fixes

[jeff-hykin]

Problem(s)

• This is a blocker for G1 rosnav stack parity with Go2, ref DIM-569
• DockerModules stopped working on dev
• configure_stream does not support different transport types (Blocker for DIM-569)
• A failed DockerModule build wouldn't stop the container (leak)
• A new container name was generated every run (huge build up of containers)
• We have no tests for docker (how it got broken on dev in the first place)
• No way to pull an image (rather than local dockerfile)
• No way to change build args (ex: can't add network flag)
• Image never rebuilds even if dockerfile edited or build args changes
• DockerModules didn't deploy in parallel
• There was a bug with error handling of WorkerManager

Closes DIM-662

Solution

• Add tests
• Minimal change to ModuleCoordinator to restore functionality: deploy docker modules in parallel
• remove configure_stream (not needed) in favor of using set_transport work as intended to allow different stream types
• stop/cleanup fixes

Breaking Changes

None

How to Test

python ./examples/docker_hello_world/hello_docker.py

Contributor License Agreement

• I have read and approved the CLA.

[greptile-apps]

Reconnect mode destroys the pre-existing container on cleanup failure

When docker_reconnect_container=True and the existing container is reached (reconnect = True), the _remove_container / docker run steps are correctly skipped. However, if _wait_for_rpc() subsequently raises (e.g. RPC timeout), the except Exception block calls _cleanup(), which unconditionally runs both docker stop <name> and docker rm -f <name> — permanently destroying the container the caller wanted to preserve.

# _cleanup() always does:
_run([_docker_bin(self.config), "stop", self._container_name], ...)
_remove_container(self.config, self._container_name)

A user who sets docker_reconnect_container=True almost certainly wants the container to survive a failed reconnect attempt. Consider tracking whether __init__ created the container itself (vs reconnected to one that was already running) and only stopping/removing it in _cleanup() when DockerModule was responsible for starting it:

self._owns_container: bool = not reconnect   # set just before rpc.start()

# _cleanup():
if self._owns_container:
    with suppress(Exception):
        _run([_docker_bin(self.config), "stop", self._container_name], ...)
    with suppress(Exception):
        _remove_container(self.config, self._container_name)

[greptile-apps]

start_all_modules silently discards all-but-first exception

executor.map returns an ordered iterator; list() re-raises the first exception it encounters and discards all later ones. When two or more module.start() calls fail concurrently (e.g. two Docker RPC calls time out), only the first failure is surfaced to the caller — the rest are silently swallowed, making diagnosis very hard.

DockerModule.start() does call self.stop() on its own failure, so containers are cleaned up individually, but the exception itself is lost.

Consider replacing with safe_thread_map (already available in this PR) and an on_errors handler that re-raises an ExceptionGroup containing all failures:

def _start_one(m):
    m.start()

def _on_start_errors(outcomes, successes, errors):
    raise ExceptionGroup("start_all_modules failed", errors)

safe_thread_map(modules, _start_one, _on_start_errors)

[greptile-apps]

_cleanup() destroys pre-existing container when docker_reconnect_container=True

This was flagged in the previous thread and the request was to recheck. The issue is still present in the current code.

When docker_reconnect_container=True is set, a pre-existing container is intentionally preserved (reconnect = True). However, if _wait_for_rpc() subsequently times out or raises, the except Exception block calls _cleanup(), which unconditionally runs docker stop <name> and docker rm -f <name>, permanently destroying the container the caller wanted to preserve.

_cleanup() has no knowledge of whether it created the container or reconnected to one. A simple fix is to track this with an instance variable:

self._owns_container: bool = False  # set to True only when DockerModule created the container

Then in _cleanup(), gate the stop/rm on self._owns_container. This way a failed reconnect attempt leaves the container running exactly as the user expects.

[greptile-apps]

docker pull runs silently — no user feedback for potentially 10-minute operation

_run() captures both stdout and stderr with capture_output=True, so the entire pull progress (layer download bars, etc.) is swallowed until either success or failure. With a default docker_pull_timeout of 600 seconds, users will see zero output for up to 10 minutes and may assume the process has hung.

Consider using subprocess.run without capture_output (or with stdout=None) so pull progress is streamed to the terminal, while still capturing stderr for the error message:

result = subprocess.run(
    [config.docker_bin, "pull", config.docker_image],
    text=True,
    stderr=subprocess.PIPE,
    timeout=config.docker_pull_timeout,
)
if result.returncode != 0:
    raise RuntimeError(
        f"Failed to pull image '{config.docker_image}'.\nSTDERR:\n{result.stderr}"
    )

This matches the pattern already used in build_image() for streaming build output.