From 1dcd6d60a9cbbf67e9cafe74f1d694b7371a5612 Mon Sep 17 00:00:00 2001 From: Zach Smith Date: Thu, 15 Jan 2026 17:53:22 -0800 Subject: [PATCH 1/2] chore: add high level docs --- docs/connector-e2e-flow.md | 83 ++++++++++++++++++++++++++++++ docs/connector-envoy-design.md | 89 ++++++++++++++++++++++++++++++++ docs/nso-connector-work.md | 92 ++++++++++++++++++++++++++++++++++ 3 files changed, 264 insertions(+) create mode 100644 docs/connector-e2e-flow.md create mode 100644 docs/connector-envoy-design.md create mode 100644 docs/nso-connector-work.md diff --git a/docs/connector-e2e-flow.md b/docs/connector-e2e-flow.md new file mode 100644 index 0000000..6bc61a9 --- /dev/null +++ b/docs/connector-e2e-flow.md @@ -0,0 +1,83 @@ +# Connector End-to-End Flow + +## Purpose +Provide a step-by-step view of the tunnel lifecycle, showing which component is +responsible for each action and what data is exchanged. + +## Actors +- **Datum Desktop / Device Agent**: runs on the user device, owns the iroh key. +- **Control Plane**: stores Connector + ConnectorAdvertisement, publishes DNS. +- **NSO (network-services-operator)**: programs Gateway API + Envoy policies. +- **Envoy Gateway**: translates Gateway API to xDS for Envoy. +- **Envoy + iroh sidecar**: handles inbound requests and dials connectors. + +## Flow +### 1) Device starts and registers a Connector +**Responsible:** Device Agent +**Data:** +- iroh NodeID (public key) +- home relay URL +- observed addresses +- device metadata/labels + +**Actions:** +- Create `Connector` (spec) for this device. +- Patch `Connector.status.connectionDetails` with NodeID + relay + addresses. + +### 2) Control plane publishes discovery records +**Responsible:** Control Plane Controller +**Data:** +- `Connector.status.connectionDetails` + +**Actions:** +- Publish DNS TXT records (or equivalent discovery data), e.g. + `_iroh...datumconnect.net`. + +### 3) Device advertises tunnels +**Responsible:** Device Agent +**Data:** +- `ConnectorAdvertisement` `spec.layer4` services (host + port) + +**Actions:** +- Create or update `ConnectorAdvertisement` linked to the Connector. + +### 4) Device creates HTTPProxy +**Responsible:** Device Agent (on behalf of the user) +**Data:** +- public hostname(s) +- backend endpoint URL (host + port) +- connector reference (name or selector) + +**Actions:** +- Create or update `HTTPProxy`. + +### 5) NSO programs Envoy +**Responsible:** NSO +**Data:** +- `HTTPProxy` backend + connector reference +- `ConnectorAdvertisement` for validation + +**Actions:** +- Validate backend is allowed by ConnectorAdvertisement. +- Create Gateway + HTTPRoute + EndpointSlice (backend points to sidecar). +- Inject metadata/headers: + - `datum-node-id` + - `datum-target-host` + - `datum-target-port` + - `datum-target-proto` +- Ensure CONNECT is enabled with `BackendTrafficPolicy` (`httpUpgrade: CONNECT`). + +### 6) Envoy handles inbound requests +**Responsible:** Envoy + iroh sidecar +**Actions:** +- Envoy routes traffic to sidecar (single listener). +- Sidecar reads metadata, dials by NodeID using discovery, and forwards traffic. + +## Data Map (Who Writes What) +- `Connector.spec`: Device Agent (creates) +- `Connector.status.connectionDetails`: Device Agent (patches) +- `ConnectorAdvertisement.spec`: Device Agent +- `HTTPProxy.spec`: Device Agent / User / UI / API client +- Gateway API + Envoy policies: NSO +- Discovery records: Control Plane Controller + diff --git a/docs/connector-envoy-design.md b/docs/connector-envoy-design.md new file mode 100644 index 0000000..4dc95a7 --- /dev/null +++ b/docs/connector-envoy-design.md @@ -0,0 +1,89 @@ +# Connector + Envoy Integration (Datum Connect) + +## Purpose +This document captures how Connector and ConnectorAdvertisement will be used to +route traffic through Envoy and a local iroh sidecar. It is intended to live in +this repo as a reference for how datum-connect participates in the overall flow. + +## Current Behavior (Datum Connect) +- A local tunnel is represented as an `Advertisment` with a TCP host/port. +- The listen node publishes a ticket to n0des for each enabled tunnel. +- The gateway extracts a subdomain, fetches the ticket, and forwards via iroh + `ConnectTunnel` to the advertised host:port. + +This is a useful baseline, but connectors move discovery and advertisement into +the control plane. + +## Target Behavior (Connector-based) +### Control Plane Resources +- **Connector**: represents a running connector instance and its capabilities. + - Connection details (public key, relay, addresses) are stored in + `status.connectionDetails`. + - The device agent writes `status.connectionDetails`; the control plane + controller consumes it to publish discovery records. +- **ConnectorAdvertisement**: describes the destinations reachable via the + connector. + - For MVP, we only use `spec.layer4` (TCP/UDP services). + +### Envoy + Sidecar Data Path +1) Envoy receives a request from the client. +2) Envoy routes the request to a local sidecar backend (not the real target). +3) The sidecar reads per-request metadata that identifies: + - Connector NodeID (iroh public key) + - Destination host + - Destination port + - Destination protocol (tcp/udp/ip) +4) The sidecar dials the remote connector using iroh and forwards traffic. + +## Sidecar Responsibilities +The iroh sidecar running alongside Envoy must: +- Maintain an iroh endpoint and connectivity state. +- Accept local connections from Envoy (HTTP CONNECT, CONNECT-UDP, CONNECT-IP). +- Resolve the connector NodeID to connection details from the control plane + (public key, relay, addresses) or dial by NodeID via discovery. +Potentially(todo(zach): find out where else this might be enforced): +- Enforce the allowed destination set from ConnectorAdvertisement. +- Gate functionality by connector capability status. + +## How Envoy is Programmed +NSO will convert Datum CRDs into Gateway API resources, then apply +EnvoyPatchPolicies to inject per-route metadata for the sidecar. The key idea: +the EndpointSlice points to the local sidecar, not the real backend. +CONNECT must be explicitly enabled on the Gateway via +`gateway.envoyproxy.io/v1alpha1 BackendTrafficPolicy` with `httpUpgrade: CONNECT`. +With that enabled, a single HTTP listener can serve all tunnels with per-route +metadata/header injection. + +### Route Metadata Contract (proposed) +Envoy will attach either request headers or route metadata for the sidecar to +consume. For example: +- `datum-node-id`: Connector NodeID (iroh public key) +- `datum-target-host`: backend host +- `datum-target-port`: backend port +- `datum-target-proto`: tcp | udp + +These values are derived from HTTPProxy backend `endpoint` and connector +references. + +## Discovery Publishing +The device agent updates `status.connectionDetails` with its NodeID (public key), +home relay, and observed addresses. A control plane controller publishes DNS TXT +records (or equivalent discovery data) from those details so peers can dial by +NodeID. + +## Validation Rules +During reconciliation: +- A backend endpoint must match a ConnectorAdvertisement entry for the selected + connector(s). +- If multiple connectors match a selector, the selection policy must be defined + (single preferred, round-robin, or fail with conflict). For MVP, choose a + single connector and emit a condition when ambiguous. +- Capability checks must pass (for example CONNECT-UDP requires capability). + +## Open Questions +- How to select among multiple connectors matching a selector. + +## Scope Note +For the initial implementation, we only support Layer 4 advertisements and +CONNECT-TCP. Layer 3 CIDRs and CONNECT-IP are deferred. + diff --git a/docs/nso-connector-work.md b/docs/nso-connector-work.md new file mode 100644 index 0000000..f0d1a43 --- /dev/null +++ b/docs/nso-connector-work.md @@ -0,0 +1,92 @@ +# NSO Work Plan: Connector + Envoy Programming + +## Goal +Add Connector and ConnectorAdvertisement support to the +network-services-operator (NSO) so HTTPProxy backends can be routed through a +connector and the Envoy sidecar can dial via iroh. + +## Current NSO Behavior +- HTTPProxy reconciler creates Gateway + HTTPRoute + EndpointSlice from Datum + HTTPProxy. +- TrafficProtectionPolicy creates EnvoyPatchPolicy to inject Coraza filters and + metadata. +- Gateway-Envoy policy CRDs are mirrored to the downstream control plane. + +## New Controllers +### 1) Connector Controller +Responsibilities: +- Watch `Connector` resources. +- Validate connector class existence. +- The connector agent is expected to set `status.connectionDetails` and + capability conditions. +- Publish DNS TXT records (or equivalent discovery records) from + `status.connectionDetails` for dial-by-NodeID. + +### 2) ConnectorAdvertisement Controller +Responsibilities: +- Watch `ConnectorAdvertisement` resources. +- Validate `connectorRef` exists and is in the same namespace. +- Track advertisements for later HTTPProxy reconciliation. + +## HTTPProxy Reconciliation Changes +### Selection +- Add support for `backend.connector` references in the Datum HTTPProxy API. +- Resolve connector by name or label selector. +- If multiple connectors match, apply a deterministic selection rule (or fail + with a condition for MVP). + +### Validation +- Ensure the backend endpoint matches an advertised L4 service in + ConnectorAdvertisement. +- Validate protocol and port are allowed. +- Validate required connector capability (CONNECT-TCP/UDP). + +### Gateway API Programming +- Keep Gateway + HTTPRoute creation as-is. +- For connector-backed routes, replace EndpointSlice host/port with the + sidecar address (for example `127.0.0.1:` or a Service backing the + sidecar). +- Ensure CONNECT is enabled on the Gateway via + `gateway.envoyproxy.io/v1alpha1 BackendTrafficPolicy` with + `httpUpgrade: CONNECT` so a single listener can serve all tunnel routes. + +### EnvoyPatchPolicy Programming +Create EnvoyPatchPolicy patches to inject metadata for the sidecar. Options: +- Add request headers (RequestHeaderModifier) on the HTTPRoute backend. + - Tested in kind with Envoy Gateway: headers were preserved for HTTP and + CONNECT once `BackendTrafficPolicy` enabled CONNECT. +- Add per-route metadata via JSONPatch (RouteConfiguration). + +Recommended metadata keys: +- `datum-node-id` +- `datum-target-host` +- `datum-target-port` +- `datum-target-proto` + +These values come from the original backend endpoint and the chosen connector. +The node ID is the connector's iroh public key (NodeID). + +## Required API Updates +- Extend `HTTPProxyRuleBackend` to include `connector` (name or selector). +- Add validation to reject invalid combinations (for example connector + + non-connectable protocols). + +## Downstream Expectations +Envoy Gateway continues to watch Gateway API resources and EnvoyPatchPolicy. +NSO remains responsible for: +- Producing correct EndpointSlices +- Injecting metadata for the sidecar +- Surfacing errors via HTTPProxy conditions + +## MVP Checklist +- Add `connector` fields to HTTPProxy API and validation. +- Implement Connector + ConnectorAdvertisement controllers. +- Update HTTPProxy controller to handle connector-backed routes: + - select connector + - validate advertisement + - swap EndpointSlice destination to sidecar + - attach metadata via EnvoyPatchPolicy or request headers +- Ensure BackendTrafficPolicy enables CONNECT on the Gateway. +- Define selection policy for multiple connector matches. +- Add conditions to HTTPProxy status for selection/validation errors. + From ec92927c1ae19598668518e9c1bed2b929837fd8 Mon Sep 17 00:00:00 2001 From: Zach Smith Date: Thu, 15 Jan 2026 19:37:52 -0800 Subject: [PATCH 2/2] chore: fix connector flow doc formatting --- docs/connector-e2e-flow.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/connector-e2e-flow.md b/docs/connector-e2e-flow.md index 6bc61a9..ea0f1f1 100644 --- a/docs/connector-e2e-flow.md +++ b/docs/connector-e2e-flow.md @@ -14,6 +14,7 @@ responsible for each action and what data is exchanged. ## Flow ### 1) Device starts and registers a Connector **Responsible:** Device Agent + **Data:** - iroh NodeID (public key) - home relay URL @@ -26,6 +27,7 @@ responsible for each action and what data is exchanged. ### 2) Control plane publishes discovery records **Responsible:** Control Plane Controller + **Data:** - `Connector.status.connectionDetails` @@ -35,6 +37,7 @@ responsible for each action and what data is exchanged. ### 3) Device advertises tunnels **Responsible:** Device Agent + **Data:** - `ConnectorAdvertisement` `spec.layer4` services (host + port) @@ -43,6 +46,7 @@ responsible for each action and what data is exchanged. ### 4) Device creates HTTPProxy **Responsible:** Device Agent (on behalf of the user) + **Data:** - public hostname(s) - backend endpoint URL (host + port) @@ -53,6 +57,7 @@ responsible for each action and what data is exchanged. ### 5) NSO programs Envoy **Responsible:** NSO + **Data:** - `HTTPProxy` backend + connector reference - `ConnectorAdvertisement` for validation @@ -69,6 +74,7 @@ responsible for each action and what data is exchanged. ### 6) Envoy handles inbound requests **Responsible:** Envoy + iroh sidecar + **Actions:** - Envoy routes traffic to sidecar (single listener). - Sidecar reads metadata, dials by NodeID using discovery, and forwards traffic.