Skip to content

infra: Hetzner server quota hit during E2E — claude and openclaw fail in batch 1 when spawn-szil is running #3111

@la14-1

Description

@la14-1

Summary

During E2E quality sweep (2026-03-30), Hetzner E2E dropped from 8/9 PASS to 6/9 PASS. The new failures are claude and openclaw in batch 1.

Root Cause

The Hetzner account has a low server/Primary IP quota. With spawn-szil (created 2026-03-27) already running, the E2E batch 1 creates 3 servers in parallel (zeroclaw + claude + openclaw). Zeroclaw wins the race and gets provisioned; claude and openclaw then hit the server limit:

Hetzner API error (HTTP 403): server_limit_reached
Your Hetzner account has reached its Primary IP limit.

Evidence

  • spawn-szil (created 2026-03-27) is a pre-existing server that reduces available quota
  • Batch 1 runs 3 agents in parallel — zeroclaw succeeds, the other 2 fail
  • Subsequent batches (codex, opencode, kilocode, hermes, cursor, junie) all PASS because zeroclaw was torn down, freeing quota

Impact

Fix Options

  1. Delete spawn-szil if it's no longer needed — this frees 1 quota slot and likely restores 8/9 PASS
  2. Request a quota increase at https://console.hetzner.cloud/limits
  3. Reduce Hetzner E2E parallelism from 3 to 2 so batch 1 only creates 2 servers

-- qa/e2e-tester

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingin-progressIssue is being actively worked on

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions