chore: add high level docs

docs/connector-e2e-flow.md

@@ -0,0 +1,89 @@
+# Connector End-to-End Flow
+
+## Purpose
+Provide a step-by-step view of the tunnel lifecycle, showing which component is
+responsible for each action and what data is exchanged.
+
+## Actors
+- **Datum Desktop / Device Agent**: runs on the user device, owns the iroh key.
+- **Control Plane**: stores Connector + ConnectorAdvertisement, publishes DNS.
+- **NSO (network-services-operator)**: programs Gateway API + Envoy policies.
+- **Envoy Gateway**: translates Gateway API to xDS for Envoy.
+- **Envoy + iroh sidecar**: handles inbound requests and dials connectors.
+
+## Flow
+### 1) Device starts and registers a Connector
+**Responsible:** Device Agent
+
+**Data:**
+- iroh NodeID (public key)
+- home relay URL
+- observed addresses
+- device metadata/labels
+
+**Actions:**
+- Create `Connector` (spec) for this device.
+- Patch `Connector.status.connectionDetails` with NodeID + relay + addresses.
+
+### 2) Control plane publishes discovery records
+**Responsible:** Control Plane Controller
+
+**Data:**
+- `Connector.status.connectionDetails`
+
+**Actions:**
+- Publish DNS TXT records (or equivalent discovery data), e.g.
+  `_iroh.<node-id>.<project>.datumconnect.net`.
+
+### 3) Device advertises tunnels
+**Responsible:** Device Agent
+
+**Data:**
+- `ConnectorAdvertisement` `spec.layer4` services (host + port)
+
+**Actions:**
+- Create or update `ConnectorAdvertisement` linked to the Connector.
+
+### 4) Device creates HTTPProxy
+**Responsible:** Device Agent (on behalf of the user)
+
+**Data:**
+- public hostname(s)
+- backend endpoint URL (host + port)
+- connector reference (name or selector)
+
+**Actions:**
+- Create or update `HTTPProxy`.
+
+### 5) NSO programs Envoy
+**Responsible:** NSO
+
+**Data:**
+- `HTTPProxy` backend + connector reference
+- `ConnectorAdvertisement` for validation
+
+**Actions:**
+- Validate backend is allowed by ConnectorAdvertisement.
+- Create Gateway + HTTPRoute + EndpointSlice (backend points to sidecar).
+- Inject metadata/headers:
+  - `datum-node-id`
+  - `datum-target-host`
+  - `datum-target-port`
+  - `datum-target-proto`
+- Ensure CONNECT is enabled with `BackendTrafficPolicy` (`httpUpgrade: CONNECT`).
+
+### 6) Envoy handles inbound requests
+**Responsible:** Envoy + iroh sidecar
+
+**Actions:**
+- Envoy routes traffic to sidecar (single listener).
+- Sidecar reads metadata, dials by NodeID using discovery, and forwards traffic.
+
+## Data Map (Who Writes What)
+- `Connector.spec`: Device Agent (creates)
+- `Connector.status.connectionDetails`: Device Agent (patches)
+- `ConnectorAdvertisement.spec`: Device Agent
+- `HTTPProxy.spec`: Device Agent / User / UI / API client
+- Gateway API + Envoy policies: NSO
+- Discovery records: Control Plane Controller
+

docs/connector-envoy-design.md

@@ -0,0 +1,89 @@
+# Connector + Envoy Integration (Datum Connect)
+
+## Purpose
+This document captures how Connector and ConnectorAdvertisement will be used to
+route traffic through Envoy and a local iroh sidecar. It is intended to live in
+this repo as a reference for how datum-connect participates in the overall flow.
+
+## Current Behavior (Datum Connect)
+- A local tunnel is represented as an `Advertisment` with a TCP host/port.
+- The listen node publishes a ticket to n0des for each enabled tunnel.
+- The gateway extracts a subdomain, fetches the ticket, and forwards via iroh
+  `ConnectTunnel` to the advertised host:port.
+
+This is a useful baseline, but connectors move discovery and advertisement into
+the control plane.
+
+## Target Behavior (Connector-based)
+### Control Plane Resources
+- **Connector**: represents a running connector instance and its capabilities.
+  - Connection details (public key, relay, addresses) are stored in
+    `status.connectionDetails`.
+  - The device agent writes `status.connectionDetails`; the control plane
+    controller consumes it to publish discovery records.
+- **ConnectorAdvertisement**: describes the destinations reachable via the
+  connector.
+  - For MVP, we only use `spec.layer4` (TCP/UDP services).
+
+### Envoy + Sidecar Data Path
+1) Envoy receives a request from the client.
+2) Envoy routes the request to a local sidecar backend (not the real target).
+3) The sidecar reads per-request metadata that identifies:
+   - Connector NodeID (iroh public key)
+   - Destination host
+   - Destination port
+   - Destination protocol (tcp/udp/ip)
+4) The sidecar dials the remote connector using iroh and forwards traffic.
+
+## Sidecar Responsibilities
+The iroh sidecar running alongside Envoy must:
+- Maintain an iroh endpoint and connectivity state.
+- Accept local connections from Envoy (HTTP CONNECT, CONNECT-UDP, CONNECT-IP).
+- Resolve the connector NodeID to connection details from the control plane
+  (public key, relay, addresses) or dial by NodeID via discovery.
+Potentially(todo(zach): find out where else this might be enforced):
+- Enforce the allowed destination set from ConnectorAdvertisement.
+- Gate functionality by connector capability status.
+
+## How Envoy is Programmed
+NSO will convert Datum CRDs into Gateway API resources, then apply
+EnvoyPatchPolicies to inject per-route metadata for the sidecar. The key idea:
+the EndpointSlice points to the local sidecar, not the real backend.
+CONNECT must be explicitly enabled on the Gateway via
+`gateway.envoyproxy.io/v1alpha1 BackendTrafficPolicy` with `httpUpgrade: CONNECT`.
+With that enabled, a single HTTP listener can serve all tunnels with per-route
+metadata/header injection.
+
+### Route Metadata Contract (proposed)
+Envoy will attach either request headers or route metadata for the sidecar to
+consume. For example:
+- `datum-node-id`: Connector NodeID (iroh public key)
+- `datum-target-host`: backend host
+- `datum-target-port`: backend port
+- `datum-target-proto`: tcp | udp
+
+These values are derived from HTTPProxy backend `endpoint` and connector
+references.
+
+## Discovery Publishing
+The device agent updates `status.connectionDetails` with its NodeID (public key),
+home relay, and observed addresses. A control plane controller publishes DNS TXT
+records (or equivalent discovery data) from those details so peers can dial by
+NodeID.
+
+## Validation Rules
+During reconciliation:
+- A backend endpoint must match a ConnectorAdvertisement entry for the selected
+  connector(s).
+- If multiple connectors match a selector, the selection policy must be defined
+  (single preferred, round-robin, or fail with conflict). For MVP, choose a
+  single connector and emit a condition when ambiguous.
+- Capability checks must pass (for example CONNECT-UDP requires capability).
+
+## Open Questions
+- How to select among multiple connectors matching a selector.
+
+## Scope Note
+For the initial implementation, we only support Layer 4 advertisements and
+CONNECT-TCP. Layer 3 CIDRs and CONNECT-IP are deferred.
+

docs/nso-connector-work.md

@@ -0,0 +1,92 @@
+# NSO Work Plan: Connector + Envoy Programming
+
+## Goal
+Add Connector and ConnectorAdvertisement support to the
+network-services-operator (NSO) so HTTPProxy backends can be routed through a
+connector and the Envoy sidecar can dial via iroh.
+
+## Current NSO Behavior
+- HTTPProxy reconciler creates Gateway + HTTPRoute + EndpointSlice from Datum
+  HTTPProxy.
+- TrafficProtectionPolicy creates EnvoyPatchPolicy to inject Coraza filters and
+  metadata.
+- Gateway-Envoy policy CRDs are mirrored to the downstream control plane.
+
+## New Controllers
+### 1) Connector Controller
+Responsibilities:
+- Watch `Connector` resources.
+- Validate connector class existence.
+- The connector agent is expected to set `status.connectionDetails` and
+  capability conditions.
+- Publish DNS TXT records (or equivalent discovery records) from
+  `status.connectionDetails` for dial-by-NodeID.
+
+### 2) ConnectorAdvertisement Controller
+Responsibilities:
+- Watch `ConnectorAdvertisement` resources.
+- Validate `connectorRef` exists and is in the same namespace.
+- Track advertisements for later HTTPProxy reconciliation.
+
+## HTTPProxy Reconciliation Changes
+### Selection
+- Add support for `backend.connector` references in the Datum HTTPProxy API.
+- Resolve connector by name or label selector.
+- If multiple connectors match, apply a deterministic selection rule (or fail
+  with a condition for MVP).
+
+### Validation
+- Ensure the backend endpoint matches an advertised L4 service in
+  ConnectorAdvertisement.
+- Validate protocol and port are allowed.
+- Validate required connector capability (CONNECT-TCP/UDP).
+
+### Gateway API Programming
+- Keep Gateway + HTTPRoute creation as-is.
+- For connector-backed routes, replace EndpointSlice host/port with the
+  sidecar address (for example `127.0.0.1:<port>` or a Service backing the
+  sidecar).
+- Ensure CONNECT is enabled on the Gateway via
+  `gateway.envoyproxy.io/v1alpha1 BackendTrafficPolicy` with
+  `httpUpgrade: CONNECT` so a single listener can serve all tunnel routes.
+
+### EnvoyPatchPolicy Programming
+Create EnvoyPatchPolicy patches to inject metadata for the sidecar. Options:
+- Add request headers (RequestHeaderModifier) on the HTTPRoute backend.
+  - Tested in kind with Envoy Gateway: headers were preserved for HTTP and
+    CONNECT once `BackendTrafficPolicy` enabled CONNECT.
+- Add per-route metadata via JSONPatch (RouteConfiguration).
+
+Recommended metadata keys:
+- `datum-node-id`
+- `datum-target-host`
+- `datum-target-port`
+- `datum-target-proto`
+
+These values come from the original backend endpoint and the chosen connector.
+The node ID is the connector's iroh public key (NodeID).
+
+## Required API Updates
+- Extend `HTTPProxyRuleBackend` to include `connector` (name or selector).
+- Add validation to reject invalid combinations (for example connector +
+  non-connectable protocols).
+
+## Downstream Expectations
+Envoy Gateway continues to watch Gateway API resources and EnvoyPatchPolicy.
+NSO remains responsible for:
+- Producing correct EndpointSlices
+- Injecting metadata for the sidecar
+- Surfacing errors via HTTPProxy conditions
+
+## MVP Checklist
+- Add `connector` fields to HTTPProxy API and validation.
+- Implement Connector + ConnectorAdvertisement controllers.
+- Update HTTPProxy controller to handle connector-backed routes:
+  - select connector
+  - validate advertisement
+  - swap EndpointSlice destination to sidecar
+  - attach metadata via EnvoyPatchPolicy or request headers
+- Ensure BackendTrafficPolicy enables CONNECT on the Gateway.
+- Define selection policy for multiple connector matches.
+- Add conditions to HTTPProxy status for selection/validation errors.
+