|
1 | | -# Ponos |
| 1 | +# Page |
2 | 2 |
|
3 | | -**Ponos** (Greek: Πόνος) means *toil*, *labor*, or *sustained effort*. |
4 | | - |
5 | | -Ponos takes on that toil for node operators. It is the terminal interface for **nodeoperator.ai**, an autonomous nodeoperator agent that deploys, manages, and remediates issues with blockchain infrastructures through GitHub-based workflows. |
6 | | - |
7 | | -## Interaction Interfaces |
8 | | - |
9 | | -- **TUI** (Terminal UI) - Interactive terminal interface |
10 | | -- **Slack** - Slash commands and bot integration |
11 | | -- **Coming Soon**: GitHub Actions, Discord, GitHub Comments |
12 | | - |
13 | | -## Installation |
14 | | - |
15 | | -### Quick install (recommended) |
16 | | - |
17 | | -```bash |
18 | | -curl -sSL https://nodeoperator.ai/install.sh | sh |
19 | | -``` |
20 | | - |
21 | | -### Build from source |
22 | | - |
23 | | -```bash |
24 | | -git clone https://github.com/blockops-sh/ponos |
25 | | -cd ponos |
26 | | -make build-ponos |
27 | | -``` |
28 | | - |
29 | | -The binary is produced at `./bin/ponos`. |
30 | | - |
31 | | -### Docker |
32 | | - |
33 | | -```bash |
34 | | -docker pull ghcr.io/blockopsnetwork/ponos-server:latest |
35 | | -``` |
36 | | - |
37 | | -## Usage |
38 | | - |
39 | | -Ponos supports two main workflows: |
40 | | - |
41 | | -### 1. Upgrading Blockchain Clients and Components |
42 | | - |
43 | | -**Supported Networks:** |
44 | | -- **Ethereum** (execution clients: Geth, Nethermind, Besu, Erigon; consensus clients: Lighthouse, Prysm, Teku, Nimbus) |
45 | | -- **EVM-compatible chains** (Polygon, Arbitrum, Optimism, Base, etc.) |
46 | | -- **Polkadot Relaychains** (Polkadot, Kusama) |
47 | | -- **Cosmos chains** (experimental support) |
48 | | -- **Solana** (experimental support) |
49 | | - |
50 | | -*Note: While we technically support all EVM chains and clients, testing has been primarily focused on Ethereum execution and consensus clients.* |
51 | | - |
52 | | -**TUI Interface** |
53 | | -```bash |
54 | | -./bin/ponos |
55 | | -``` |
56 | | -Once in the TUI, use prompts like: |
57 | | -- `"Upgrade mainnet Geth to the latest version"` |
58 | | -- `"Show me available Lighthouse versions for testnet"` |
59 | | -- `"Create a PR to update Polygon nodes to v1.2.3"` |
60 | | -- `"What's the current version of Besu on mainnet?"` |
61 | | - |
62 | | -**Slack Interface** |
63 | | -``` |
64 | | -@ponos (replace with your slacp app name) <your prompt> |
65 | | -``` |
66 | | -Example Slack messages: |
67 | | -- `@ponos upgrade mainnet Geth to the latest version` |
68 | | -- `@ponos create a PR to update Polygon nodes to v1.2.3` |
69 | | -- `@ponos show me the current Besu version on mainne` |
70 | | -- `@ponos what's the recent Prysm upgrade about? Do I need to make any configurational changes? If so, create a PR to upgrade and make the chnages` |
71 | | - |
72 | | -### 2. Diagnosing Blockchain Nodes on Kubernetes |
73 | | - |
74 | | -**TUI Interface** |
75 | | -```bash |
76 | | -./bin/ponos |
77 | | -``` |
78 | | -Once in the TUI, use prompts like: |
79 | | -- `"Diagnose mainnet Ethereum nodes"` |
80 | | -- `"Check why Geth pods are failing on testnet"` |
81 | | -- `"Show me resource usage for Lighthouse validators"` |
82 | | -- `"Analyze logs for polygon-bor nodes in the last hour"` |
83 | | -- `"Why is my consensus client out of sync?"` |
84 | | - |
85 | | -**Slack Interface** |
86 | | -```bash |
87 | | -/diagnose <service or prompt> |
88 | | -``` |
89 | | -Example Slack commands: |
90 | | -- `/diagnose mainnet-geth` - Quick health check for specific service |
91 | | -- `/diagnose check mainnet Ethereum nodes health` |
92 | | -- `/diagnose why are my Geth pods failing on testnet?` |
93 | | - |
94 | | -Or use @ponos for more complex requests: |
95 | | -- `@ponos show resource usage for Lighthouse validators` |
96 | | -- `@ponos analyze logs for polygon-bor nodes in the last hour` |
97 | | - |
98 | | -**Server Mode (for Slack + webhooks)** |
99 | | -```bash |
100 | | -make run-ponos |
101 | | -``` |
102 | | -- Enables Slack slash commands and GitHub webhook listening |
103 | | -- Handles background diagnostic tasks |
104 | | -- (Note: Will be migrating to agent-core soon) |
105 | | - |
106 | | -## Configuration |
107 | | - |
108 | | -Configuration is provided via `ponos.yml` file. Copy `ponos.yml.example` to `ponos.yml` and customize for your environment. |
109 | | - |
110 | | -### Configuration Structure |
111 | | - |
112 | | -```yaml |
113 | | -version: 1 |
114 | | - |
115 | | -api_endpoint: "https://api.nodeoperator.ai" |
116 | | -api_key: "your_api_key_here" |
117 | | - |
118 | | -integrations: |
119 | | - github: |
120 | | - # Choose one authentication method: |
121 | | - token: "ghp_xxx" # Personal Access Token (PAT) |
122 | | - # OR GitHub App (recommended for production): |
123 | | - app_id: "123456" |
124 | | - install_id: "654321" |
125 | | - pem_key: "/path/to/pem/key" |
126 | | - bot_name: "ponos-bot" |
127 | | - mcp_url: "http://localhost:3001" |
128 | | - |
129 | | - slack: |
130 | | - token: "xoxb-xxx" |
131 | | - signing_key: "your_slack_signing_secret" |
132 | | - verify_token: "your_slack_verification_token" |
133 | | - channel: "sre-tasks" |
134 | | - |
135 | | - # Telescope observability (for node metrics, logs, and blockchain client metrics - see https://github.com/blockopsnetwork/telescope) |
136 | | - telescope: |
137 | | - project_id: "" |
138 | | - project_name: "your-project-name" |
139 | | - prometheus_url: "https://your-prometheus-endpoint" |
140 | | - prometheus_username: "" |
141 | | - prometheus_password: "" |
142 | | - loki_url: "https://your-loki-endpoint" |
143 | | - loki_username: "" |
144 | | - loki_password: "" |
145 | | - |
146 | | -# Diagnostics & monitoring (optional) |
147 | | -diagnostics: |
148 | | - enabled: false |
149 | | - provider: "telescope" # Options: telescope, kubernetes |
150 | | - github: |
151 | | - owner: "your-org" |
152 | | - repo: "infrastructure-issues" |
153 | | - slack: |
154 | | - channel: "sre-tasks" |
155 | | - kubernetes: |
156 | | - namespace: "default" |
157 | | - resource_type: "deployment" |
158 | | - monitoring: |
159 | | - service: "your-service-name" |
160 | | - log_tail: 500 |
161 | | - eval_interval: 2 |
162 | | - |
163 | | -# Server settings |
164 | | -server: |
165 | | - port: "8080" |
166 | | - enable_release_listener: false |
167 | | -``` |
168 | | -
|
169 | | -### GitHub Authentication |
170 | | -
|
171 | | -**Personal Access Token (PAT)** |
172 | | -- Simpler setup for development and personal use |
173 | | -- Uses your personal GitHub account credentials |
174 | | -- Requires `repo` scope for private repositories |
175 | | -- Set `integrations.github.token` only |
176 | | - |
177 | | -**GitHub App (Recommended)** |
178 | | -- Recommended for production and organizational use |
179 | | -- More secure with fine-grained permissions |
180 | | -- Operations appear as the configured bot name |
181 | | -- Set `integrations.github.app_id`, `install_id`, and `pem_key` |
182 | | - |
183 | | -### Telescope Integration |
184 | | - |
185 | | -Telescope provides observability for blockchain nodes, including system metrics (CPU, memory, disk), blockchain client metrics (sync status, peers, attestations), and container logs. |
186 | | - |
187 | | -| Field | Description | |
188 | | -|-------|-------------| |
189 | | -| `project_id` | Telescope project identifier | |
190 | | -| `project_name` | Project name for multi-tenant scoping | |
191 | | -| `prometheus_url` | Prometheus/Thanos endpoint for metrics | |
192 | | -| `prometheus_username` | Basic auth username | |
193 | | -| `prometheus_password` | Basic auth password | |
194 | | -| `loki_url` | Loki endpoint for logs | |
195 | | -| `loki_username` | Basic auth username | |
196 | | -| `loki_password` | Basic auth password | |
197 | | - |
198 | | -See [Telescope documentation](https://github.com/blockopsnetwork/telescope) for setup. |
199 | | - |
200 | | -### Diagnostics Provider |
201 | | - |
202 | | -The `diagnostics.provider` field determines where Ponos fetches node metrics and logs: |
203 | | - |
204 | | -| Provider | Description | |
205 | | -|----------|-------------| |
206 | | -| `telescope` | Uses Telescope's Prometheus and Loki endpoints for metrics and logs | |
207 | | -| `kubernetes` | Queries Kubernetes API directly for pod logs and resource metrics | |
208 | | - |
209 | | -### Optional Environment Variables |
210 | | - |
211 | | -```bash |
212 | | -PONOS_TUI_LOG_PATH=/tmp/ponos.log |
213 | | -``` |
214 | | - |
215 | | -### GitHub Webhook Settings |
216 | | - |
217 | | -For automated release detection and pull request creation: |
218 | | - |
219 | | -Set to 'true' to enable GitHub webhook listener |
220 | | -```bash |
221 | | -ENABLE_RELEASE_LISTENER=false |
222 | | -``` |
223 | | - |
224 | | -**Release Webhook Listener** (`ENABLE_RELEASE_LISTENER`) |
225 | | -- **Default**: `false` (disabled) |
226 | | -- **Purpose**: Automatically creates upgrade PRs when new blockchain client releases are detected |
227 | | -- **When enabled**: Ponos server listens for GitHub release webhooks at `/webhooks/releases` |
228 | | -- **Workflow**: |
229 | | - 1. External systems send release notifications to Ponos webhook endpoint |
230 | | - 2. Ponos analyzes the release using NodeOperator.ai agent |
231 | | - 3. Automatically creates pull requests with version updates and deployment changes |
232 | | - 4. Sends summary notifications to configured Slack channel |
233 | | - |
234 | | -**Setup Requirements** (when enabled): |
235 | | -- Ponos must be running in server mode (`make run-ponos`) |
236 | | -- Webhook endpoint must be publicly accessible (consider using ngrok for development) |
237 | | -- Subscribe to release notifications via **Relay** at [relay.blockops.network](https://relay.blockops.network) |
238 | | - - Relay is our specialized service for blockchain release monitoring |
239 | | - - Configure your webhook endpoint: `https://your-ponos-server.com/webhooks/releases` |
240 | | - - Subscribe to specific blockchain networks you want to monitor |
241 | | - - Relay will automatically send release payloads when new versions are published |
242 | | - |
243 | | -**Note**: This feature is primarily for advanced automation setups. Most users can rely on manual upgrade commands via TUI or Slack interface. |
244 | | - |
245 | | -### MCP Server Configuration |
246 | | - |
247 | | -#### Default Mode (Hosted MCP Servers) |
248 | | -- By default, Ponos uses MCP servers hosted by NodeOperator.ai |
249 | | -- Default endpoints: |
250 | | - - GitHub MCP: `github-mcp.nodeoperator.ai` |
251 | | - - Slack MCP: `slack-mcp.nodeoperator.ai` |
252 | | - - Kubernetes MCP: `k8s-mcp.nodeoperator.ai` |
253 | | -- Source code: [github.com/blockopsnetwork/mcp-servers](https://github.com/blockopsnetwork/mcp-servers) |
254 | | -- No additional MCP server setup required |
255 | | - |
256 | | -#### Bring Your Own MCP Servers (BYOM) |
257 | | -To use your own MCP servers, set these environment variables: |
258 | | - |
259 | | -```bash |
260 | | -# Override default hosted MCP servers |
261 | | -GITHUB_MCP_URL=http://your-github-mcp-server:3001 |
262 | | -SLACK_MCP_URL=http://your-slack-mcp-server:3002 |
263 | | -K8S_MCP_URL=http://your-k8s-mcp-server:3003 |
264 | | -``` |
265 | | - |
266 | | -**GitHub MCP Server** |
267 | | -- Handles GitHub API calls, branch creation, and pull requests |
268 | | -- Required for all repository operations |
269 | | -- Default: Uses hosted GitHub MCP server |
270 | | - |
271 | | -**Slack Integration** |
272 | | -- Built directly into Ponos using official Slack Go SDK |
273 | | -- No separate Slack MCP server needed unless using custom implementation |
274 | | - |
275 | | -## Repository Configuration |
276 | | - |
277 | | -Ponos requires a configuration file to discover and manage your blockchain deployments. This file tells Ponos which GitHub repositories contain your node deployment configurations. |
278 | | - |
279 | | -### Setup |
280 | | - |
281 | | -1. Copy the example configuration: |
282 | | - ```bash |
283 | | - cp repo-config.yaml.example repo-config.yaml |
284 | | - ``` |
285 | | - |
286 | | -2. Set the configuration file path: |
287 | | - ```bash |
288 | | - export CONFIG_YAML_PATH=repo-config.yaml |
289 | | - ``` |
290 | | - |
291 | | -### Configuration Format |
292 | | - |
293 | | -```yaml |
294 | | -version: 1 |
295 | | -projects: |
296 | | - - network: ethereum |
297 | | - project_name: eth-consensus-client |
298 | | - owner: your-org |
299 | | - name: ethereum-deployments |
300 | | - branch: main |
301 | | - paths: |
302 | | - - "deployments/prysm/values.yaml" |
303 | | - - "deployments/lighthouse/values.yaml" |
304 | | - - "k8s/ethereum/prysm-deployment.yaml" |
305 | | -
|
306 | | - - network: ethereum |
307 | | - project_name: eth-execution-client |
308 | | - owner: your-org |
309 | | - name: ethereum-deployments |
310 | | - branch: main |
311 | | - paths: |
312 | | - - "deployments/geth/values.yaml" |
313 | | - - "deployments/nethermind/values.yaml" |
314 | | - - "k8s/ethereum/geth-deployment.yaml" |
315 | | -
|
316 | | - - network: arbitrum |
317 | | - project_name: arbitrum-node |
318 | | - owner: your-org |
319 | | - name: layer2-infrastructure |
320 | | - branch: main |
321 | | - paths: |
322 | | - - "arbitrum/nitro/values.yaml" |
323 | | - - "k8s/arbitrum-deployment.yaml" |
324 | | -``` |
325 | | - |
326 | | -### Configuration Fields |
327 | | - |
328 | | -| Field | Required | Description | |
329 | | -|-------|----------|-------------| |
330 | | -| `network` | Yes | Network name (ethereum, polkadot, arbitrum, etc.) | |
331 | | -| `project_name` | Yes | Unique identifier for this deployment project | |
332 | | -| `owner` | Yes | GitHub organization or username | |
333 | | -| `name` | Yes | Repository name containing deployment configs | |
334 | | -| `branch` | No | Git branch for updates (default: "main") | |
335 | | -| `paths` | Yes | List of file paths containing node configurations | |
336 | | - |
337 | | -### Supported File Types |
338 | | - |
339 | | -Ponos can update Docker image tags in various configuration formats: |
340 | | - |
341 | | -- **Kubernetes YAML** - Deployment, StatefulSet, DaemonSet manifests |
342 | | -- **Helm values.yaml** - Chart configuration files |
343 | | -- **Docker Compose** - docker-compose.yml files |
344 | | -- **Custom configs** - Any YAML with `image:` or `tag:` fields |
345 | | - |
346 | | - |
347 | | -## Development |
348 | | - |
349 | | -```bash |
350 | | -go test ./... |
351 | | -make build-ponos |
352 | | -make run-ponos |
353 | | -``` |
0 commit comments