Remote Worker deployment fails - MinIO S3 API inaccessible via Higress proxy

## 问题描述

尝试将 Worker 部署到远程服务器时，Worker 无法从 MinIO 拉取文件。MinIO 服务运行在中心服务器，但远程服务器无法通过 Higress 代理或 MinIO 直接端口访问 S3 API。

## 环境信息

| 组件 | 配置 |
|------|------|
| **HiClaw 版本** | (待确认) |
| **MinIO 版本** | 2025-09-07T16:13:09Z |
| **Higress** | Standalone 模式 |
| **端口配置** | Higress: 8080/18080, MinIO: 9000 |

## 问题现象

### Worker 启动失败日志

```
╭───── Starting ─────╮
│ CoPaw Worker       │
│ Worker: news-daily │
╰────────────────────╯
Pulling all files from MinIO...
2026-04-03 15:55:23,390 [INFO] copaw_worker.sync: mc cmd: /usr/local/bin/mc alias set hiclaw http://[SERVER_IP]:18080 admin [REDACTED]
2026-04-03 15:55:23,573 [INFO] copaw_worker.sync: mc stdout (29 chars): 'Added `hiclaw` successfully.\n'
2026-04-03 15:55:23,574 [INFO] copaw_worker.sync: mc cmd: /usr/local/bin/mc mirror hiclaw/hiclaw-storage/agents/news-daily/ /root/.copaw-worker/news-daily/ --overwrite --exclude credentials/**
2026-04-03 15:55:23,765 [WARNING] copaw_worker.sync: mirror_all: mc mirror failed: mc: <ERROR> Unable to stat source `hiclaw/hiclaw-storage/agents/news-daily/`. Object does not exist.

Failed to mirror from MinIO: Command '['/usr/local/bin/mc', 'mirror', 'hiclaw/hiclaw-storage/agents/news-daily/', '/root/.copaw-worker/news-daily/', '--overwrite', '--exclude', 'credentials/**']' returned non-zero exit status 1.
```

### MinIO 远程访问测试

**本地访问（正常）**:
```bash
# 本地 127.0.0.1:9000 - 成功
mc alias set local http://127.0.0.1:9000 admin [REDACTED]
mc ls local/hiclaw-storage/agents/news-daily/
# 输出：文件列表正常
```

**远程 Higress 代理端口（失败）**:
```bash
# 远程服务器测试
mc alias set test http://[SERVER_IP]:18080 admin [REDACTED] --api s3v4
mc ls test/hiclaw-storage/agents/news-daily/
# 输出：空（无错误但无内容）
curl -sI 'http://[SERVER_IP]:18080/hiclaw-storage/' 
# 输出：返回 Higress Console HTML 页面（不是 S3 API 响应）
```

**远程 MinIO 直接端口（失败）**:
```bash
# 远程服务器测试
mc alias set direct http://[SERVER_IP]:9000 admin [REDACTED] --api s3v4
mc ls direct/hiclaw-storage/agents/news-daily/
# 输出：The request signature we calculated does not match the signature you provided.
```

**远程 Higress 8080 端口（超时）**:
```bash
# 远程服务器测试
mc alias set test8080 http://[SERVER_IP]:8080 admin [REDACTED] --api s3v4
mc ls test8080/hiclaw-storage/agents/news-daily/
# 输出：dial tcp [SERVER_IP]:8080: i/o timeout
```

## 已尝试的解决方案

### 1. MinIO 绑定地址修复
```bash
pkill -9 minio
sleep 2
minio server /data/minio --address 0.0.0.0:9000 --console-address 0.0.0.0:9001 &
```
**结果**: 远程 9000 端口仍然签名认证失败

### 2. Higress Ingress 配置
创建 Higress Ingress 配置文件，但 Higress 运行在 standalone 模式，无法自动加载 YAML 配置。
**结果**: 无效

### 3. Higress API 配置
通过 Higress Console API 验证配置：
- Service Source: `minio` → `127.0.0.1:9000` ✅
- Route: `http-filesystem` → `minio.static:9000` ✅
- Domain: `fs-local.hiclaw.io` ✅

**结果**: 配置正确，但远程访问 8080 端口超时

### 4. Envoy 代理重启
```bash
pkill -f 'envoy -c etc/istio'
```
**结果**: 8080 端口远程访问仍然超时

## 根本原因分析

1. **网络连通性问题**: 远程服务器无法访问中心服务器的 8080 端口（i/o timeout），可能存在防火墙或安全组配置问题

2. **Higress Standalone 模式限制**: Higress Console 运行在 standalone 模式（非 Kubernetes），需要通过 Higress Console Web 界面或 API 手动配置路由

3. **MinIO Docker 代理签名问题**: MinIO 通过 Docker 代理暴露 9000 端口，远程访问时 S3 签名认证失败

4. **Higress 端口映射问题**: Envoy 监听在 8080 端口，18080 端口需要外部映射（nginx 或其他反向代理），当前 18080 端口未正确映射到 Higress

## 期望行为

1. 远程 Worker 可以通过 Higress 代理端口访问 MinIO S3 API
2. 或者远程 Worker 可以通过 MinIO 直接端口访问
3. `mc` 工具可以正常列出和同步文件

## 临时解决方案

在本地运行 Worker（不部署到远程服务器）:
```bash
copaw-worker \
  --name news-daily \
  --fs http://127.0.0.1:9000 \
  --fs-key admin \
  --fs-secret [REDACTED] \
  --console-port 8089 &
```

## 需要的帮助

1. **Higress Standalone 模式配置**: 如何通过 API 或配置文件添加 MinIO S3 路由？
2. **MinIO Docker 代理**: 如何配置 Docker 代理使远程 S3 签名认证正常工作？
3. **最佳实践**: HiClaw 官方推荐的远程 Worker 部署架构是什么？
4. **端口映射**: Higress 的 8080/18080 端口如何正确映射以支持远程访问？

## 相关文件

- `~/remote-servers/DEPLOYMENT-SUMMARY-FINAL.md`
- `~/remote-servers/MINIO-FIX.md`
- `/opt/hiclaw/scripts/init/setup-higress.sh`
- `/opt/hiclaw/scripts/lib/gateway-api.sh`


---
## Problem description

When trying to deploy a Worker to a remote server, the Worker fails to pull files from MinIO. The MinIO service runs on the central server, but remote servers cannot access the S3 API through the Higress proxy or the MinIO direct port.

## Environment information

| Components | Configuration |
|------|------|
| **HiClaw version** | (To be confirmed) |
| **MinIO version** | 2025-09-07T16:13:09Z |
| **Higress** | Standalone mode |
| **Port Configuration** | Higress: 8080/18080, MinIO: 9000 |

## Problem phenomenon

### Worker startup failure log

```
╭───── Starting ─────╮
│ CoPaw Worker │
│ Worker: news-daily │
╰────────────────────╯
Pulling all files from MinIO...
2026-04-03 15:55:23,390 [INFO] copaw_worker.sync: mc cmd: /usr/local/bin/mc alias set hiclaw http://[SERVER_IP]:18080 admin [REDACTED]
2026-04-03 15:55:23,573 [INFO] copaw_worker.sync: mc stdout (29 chars): 'Added `hiclaw` successfully.\n'
2026-04-03 15:55:23,574 [INFO] copaw_worker.sync: mc cmd: /usr/local/bin/mc mirror hiclaw/hiclaw-storage/agents/news-daily/ /root/.copaw-worker/news-daily/ --overwrite --exclude credentials/**
2026-04-03 15:55:23,765 [WARNING] copaw_worker.sync: mirror_all: mc mirror failed: mc: <ERROR> Unable to stat source `hiclaw/hiclaw-storage/agents/news-daily/`. Object does not exist.

Failed to mirror from MinIO: Command '['/usr/local/bin/mc', 'mirror', 'hiclaw/hiclaw-storage/agents/news-daily/', '/root/.copaw-worker/news-daily/', '--overwrite', '--exclude', 'credentials/**']' returned non-zero exit status 1.
```

### MinIO remote access test

**Local access (normal)**:
```bash
# local 127.0.0.1:9000 - successful
mc alias set local http://127.0.0.1:9000 admin [REDACTED]
mc ls local/hiclaw-storage/agents/news-daily/
# Output: file list is normal
```

**Remote Higress proxy port (failed)**:
```bash
# Remote server testing
mc alias set test http://[SERVER_IP]:18080 admin [REDACTED] --api s3v4
mc ls test/hiclaw-storage/agents/news-daily/
# Output: empty (no errors but no content)
curl -sI 'http://[SERVER_IP]:18080/hiclaw-storage/'
# Output: Return Higress Console HTML page (not S3 API response)
```

**Remote MinIO direct port (failed)**:
```bash
# Remote server testing
mc alias set direct http://[SERVER_IP]:9000 admin [REDACTED] --api s3v4
mc ls direct/hiclaw-storage/agents/news-daily/
# Output: The request signature we calculated does not match the signature you provided.
```

**Remote Higress port 8080 (timeout)**:
```bash
# Remote server testing
mc alias set test8080 http://[SERVER_IP]:8080 admin [REDACTED] --api s3v4
mc ls test8080/hiclaw-storage/agents/news-daily/
# Output: dial tcp [SERVER_IP]:8080: i/o timeout
```

## Tried solutions

### 1. MinIO binding address repair
```bash
pkill -9 minio
sleep 2
minio server /data/minio --address 0.0.0.0:9000 --console-address 0.0.0.0:9001 &
```
**Result**: Remote port 9000 still fails signature authentication

### 2. Higress Ingress configuration
Create Higress Ingress configuration file, but Higress runs in standalone mode and cannot automatically load YAML configuration.
**Result**: Invalid

### 3. Higress API configuration
Verify configuration via Higress Console API:
- Service Source: `minio` → `127.0.0.1:9000` ✅
- Route: `http-filesystem` → `minio.static:9000` ✅
- Domain: `fs-local.hiclaw.io` ✅

**Result**: Configuration is correct, but remote access to port 8080 times out

### 4. Envoy proxy restart
```bash
pkill -f 'envoy -c etc/istio'
```
**Result**: Remote access on port 8080 still times out

## Root cause analysis

1. **Network connectivity problem**: The remote server cannot access the 8080 port (i/o timeout) of the central server. There may be a firewall or security group configuration problem.

2. **Higress Standalone mode limitations**: Higress Console runs in standalone mode (non-Kubernetes), and routing needs to be manually configured through the Higress Console web interface or API

3. **MinIO Docker proxy signature issue**: MinIO exposes port 9000 through the Docker proxy, and S3 signature authentication fails during remote access.

4. **Higress port mapping problem**: Envoy listens on port 8080, port 18080 needs external mapping (nginx or other reverse proxy), and the current port 18080 is not correctly mapped to Higress

## Desired behavior

1. Remote Worker can access MinIO S3 API through Higress proxy port
2. Or the remote Worker can be accessed through the MinIO direct port
3. The `mc` tool can list and synchronize files normally

## Temporary solution

Run the worker locally (not deploy to remote server):
```bash
copaw-worker\
  --name news-daily \
  --fs http://127.0.0.1:9000 \
  --fs-key admin \
  --fs-secret [REDACTED] \
  --console-port 8089 &
```

## Help needed

1. **Higress Standalone Mode Configuration**: How to add MinIO S3 routing through API or configuration file?
2. **MinIO Docker Agent**: How to configure the Docker agent to make remote S3 signature authentication work properly?
3. **Best Practice**: What is HiClaw’s officially recommended remote Worker deployment architecture?
4. **Port Mapping**: How to correctly map Higress’s 8080/18080 port to support remote access?

## Related documents

- `~/remote-servers/DEPLOYMENT-SUMMARY-FINAL.md`
- `~/remote-servers/MINIO-FIX.md`
- `/opt/hiclaw/scripts/init/setup-higress.sh`
- `/opt/hiclaw/scripts/lib/gateway-api.sh`

组件	配置
HiClaw 版本	(待确认)
MinIO 版本	2025-09-07T16:13:09Z
Higress	Standalone 模式
端口配置	Higress: 8080/18080, MinIO: 9000

Components	Configuration
HiClaw version	(To be confirmed)
MinIO version	2025-09-07T16:13:09Z
Higress	Standalone mode
Port Configuration	Higress: 8080/18080, MinIO: 9000

Remote Worker deployment fails - MinIO S3 API inaccessible via Higress proxy #548

Description

问题描述

环境信息

问题现象

Worker 启动失败日志

MinIO 远程访问测试

已尝试的解决方案

1. MinIO 绑定地址修复

2. Higress Ingress 配置

3. Higress API 配置

4. Envoy 代理重启

根本原因分析

期望行为

临时解决方案

需要的帮助

相关文件

Problem description

Environment information

Problem phenomenon

Worker startup failure log

MinIO remote access test

Tried solutions

1. MinIO binding address repair

2. Higress Ingress configuration

3. Higress API configuration

4. Envoy proxy restart

Root cause analysis

Desired behavior

Temporary solution

Help needed

Related documents

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions