feat(orchestrator): unified Worker lifecycle service replacing docker-proxy

[Jing-ze]

Summary

This PR replaces the docker-proxy (a simple Docker API security proxy) with hiclaw-orchestrator — a unified Worker lifecycle service that abstracts away the underlying compute platform. The orchestrator exposes a single REST API that Manager and Workers interact with, regardless of whether workers run as local Docker containers or cloud SAE applications.

Architecture

Manager  ─── REST API ───→  Orchestrator  ───→  DockerBackend (local)
                                          ───→  SAEBackend (cloud)
                                          ───→  (future: K8s, ACS, ...)

Workers  ─── POST /credentials/sts ───→  Orchestrator  ───→  STS (scoped tokens)
         ─── POST /workers/{name}/ready ──→  Orchestrator  (readiness reporting)

What changed

Phase 1 — Restructure: Renamed docker-proxy/ to orchestrator/, restructured into proxy/, backend/, api/ packages. Defined WorkerBackend and GatewayBackend interfaces. Implemented DockerBackend. Added unified /workers/* REST API while preserving Docker API passthrough for exec/logs.

Phase 2 — Cloud backends: Implemented SAEBackend (Alibaba Cloud SAE) and APIGBackend (AI Gateway consumer management) using Go SDKs, replacing aliyun-api.py and aliyun-sae.sh. Added two-tier auth (static manager key + per-worker API keys). Added centralized STS token service — workers no longer need OIDC credentials; the orchestrator issues scoped OSS tokens via POST /credentials/sts. Worker API keys persisted to OSS for recovery across restarts.

Phase 3 — Shell simplification: Rewrote container-api.sh as a thin orchestrator API client (~170 lines, down from ~730). Simplified gateway-api.sh cloud path. Simplified create-worker.sh Step 9 into a single unified orchestrator call. Deleted aliyun-api.py (527 lines) and aliyun-sae.sh (81 lines). Removed Python SDK dependencies from Dockerfile.aliyun.

Readiness detection: SAE Create() polls until the application reaches RUNNING state. Workers self-report readiness via POST /workers/{name}/ready after agent initialization. GET /workers/{name} merges backend status with readiness: running + reported ready = ready. Unified worker_backend_wait_ready replaces Docker exec-based health polling.

Backend abstraction: WorkerBackend interface includes NeedsCredentialInjection() capability method. All backend-specific logic (credential injection, runtime env vars) is encapsulated inside each backend's Create(). Handler and main layers are fully backend-agnostic — no runtime string checks, no backend name matching. Adding a new backend (K8s, ACS) only requires implementing the interface.

Key design decisions

• Workers have no OIDC capability — orchestrator is the sole credential issuer
• HICLAW_ORCHESTRATOR_URL replaces both HICLAW_CONTAINER_API and the old HICLAW_ORCHESTRATOR_URL (unified)
• Backend selection is config-driven (presence of HICLAW_SAE_WORKER_IMAGE enables SAE), not runtime-string-driven
• STS calls use raw HTTP (no STS SDK) to minimize dependencies
• OSS V1 signing for key persistence avoids pulling in the full OSS SDK

Test plan

• cd orchestrator && go test ./... — all packages pass
• Verify grep -r '"aliyun"' orchestrator/ only appears in sae.go (backend internal)
• Verify grep -r 'IsAliyunRuntime' orchestrator/ returns 0 results
• Verify grep -r 'b.Name()' orchestrator/api/ returns 0 results
• make build-orchestrator — Docker image builds
• Cloud deployment testing with SAE workers

🤖 Generated with Claude Code

[github-actions]

❌ Integration Tests Failed

Commit: a054fbe
Workflow run: #538

Test Results

No test output captured.

Debug Log (tail)

No debug logs available.

---

📦 Download full debug logs & test artifacts

[Jing-ze]

CI failure is expected — the test-integration.yml on main still references build-docker-proxy and lacks the ORCHESTRATOR_IMAGE env var. Since PR workflows run from the base branch, this is a chicken-and-egg issue that resolves once merged.

Local make test passes correctly (all 14 integration tests).