You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/design/architecture.md
+16-23Lines changed: 16 additions & 23 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,9 +10,9 @@
10
10
11
11
## Design Principles
12
12
13
-
1.**Stateless first** — all backend services are stateless. State belongs in dedicated stores (database, cache, object store). Any service instance can be killed and replaced without impact.
13
+
1.**Stateless first** — all backend services are stateless. State belongs in dedicated stores (database, object store). Any service instance can be killed and replaced without impact.
14
14
2.**Pluggable** — cross-cutting concerns (auth, rate limiting) are middleware slots with defined interfaces. Swap implementations without touching business logic.
15
-
3.**Cloud-native** — designed for Kubernetes from day one. Service discovery via DNS, horizontal scaling per component, infrastructure and application layers are cleanly separated.
15
+
3.**Single-process standalone** — all services run in a single process via the `app` binary. The architecture is designed so that individual services *could* be split into separate deployments in the future, but the current deployment model is a single process.
16
16
17
17
## System Overview
18
18
@@ -23,17 +23,16 @@
23
23
24
24
| Component | Stateful | Description |
25
25
|-----------|----------|-------------|
26
-
|**Gateway**| No | Control-plane entry point. Reverse proxy, rate limiting, auth middleware. See [Gateway](./gateway.md). |
27
-
|**MetaService**| No | System brain. Metadata CRUD, share link lifecycle, connection token signing, service topology awareness (K8s Endpoints API). See [MetaService](./meta-service.md). |
26
+
|**Gateway**| No | Control-plane entry point. Reverse proxy for MetaService and Ingestor, rate limiting, auth middleware. Streamer is accessed directly, not through Gateway — streaming data should not pass through a reverse proxy. See [Gateway](./gateway.md). |
27
+
|**MetaService**| No | System brain. Metadata CRUD, share link lifecycle, connection token signing. See [MetaService](./meta-service.md). |
28
28
|**Ingestor**| No | Receives uploads from clients, processes into HLS. Storage is an internal detail. Stateless — no affinity needed. |
29
-
|**Streamer**| No | Serves video playback. Per-video affinity via consistent hashing for local cache optimization. Client reconnects to a different instance only on failure. |
30
-
|**Client (WASM)**| No | Core logic in Rust → WASM. Client-side load balancing, connection reuse, failover. |
29
+
|**Streamer**| No | Serves video playback. Accessed directly by clients, not proxied through Gateway. |
30
+
|**Client (WASM)**| No | Core logic in Rust → WASM. |
31
31
|**MinIO (S3)**| Yes | Object storage. Internal to Ingestor/Streamer — never exposed to clients. |
32
32
|**Database**| Yes | Metadata persistence. |
33
-
|**MemoryCache**| Yes | Rate limiting state for Gateway and MetaService. Redis, Memcached, or in-process. |
34
-
|**Queue**| Yes | Async job delivery between services (e.g. upload verification). Redis Streams, RabbitMQ, or SQS. |
33
+
|**Queue (in-memory)**| No | In-process async job delivery (e.g. upload-complete notification). Uses an in-memory channel within the single process — not a distributed message broker. |
35
34
36
-
All backend services are stateless. State belongs in dedicated stores.
35
+
All services run in a single process. Backend services are stateless; persistent state belongs in dedicated stores (database, object storage).
37
36
38
37
## User Identity
39
38
@@ -43,27 +42,21 @@ Ownership checks (e.g. "can this user delete this video?") are enforced by MetaS
43
42
44
43
## Data Plane — Connection Token Flow
45
44
46
-
Control-plane requests go through Gateway. Data-plane traffic (upload/playback) bypasses Gateway entirely — clients connect directly to Ingestor/Streamer instances.
45
+
Control-plane requests go through Gateway. Data-plane traffic (upload/playback) is served by Ingestor/Streamer directly (all within the same process).
47
46
48
-
This follows the **Kafka / Redis Cluster pattern**: client fetches a service map, computes routing locally, and connects directly to the target instance.
47
+
In the current single-process deployment, all services share the same address. The connection token flow still applies — it authorizes data-plane access regardless of deployment topology.
49
48
50
49
```
51
50
1. Client → Gateway → MetaService: "I want to upload/watch video X"
52
-
2. MetaService returns a service map (list of Ingestor/Streamer external addresses)
53
-
sourced from K8s Endpoints API, mapped to externally reachable addresses (NodePort)
54
-
3. Client computes target locally:
55
-
- Upload: round-robin across Ingestors
56
-
- Playback: consistent hash ring on video_id → pick Streamer (cache affinity)
57
-
4. Client connects directly to the chosen instance with a signed connection token
58
-
5. Upload: client sends file data to Ingestor (storage is an internal detail of Ingestor)
59
-
Playback: client requests HLS manifest/segments from Streamer (caching is an internal detail)
51
+
2. MetaService returns a connection token authorizing the action
52
+
3. Client connects to Ingestor (upload) or Streamer (playback) with the signed token
53
+
4. Upload: client sends file data to Ingestor
54
+
Playback: client requests HLS manifest/segments from Streamer
60
55
```
61
56
62
-
**On failure**: client refreshes the service map from MetaService, recomputes target, reconnects. Only the failed node's connections are affected — consistent hashing minimizes cache disruption on Streamer topology changes.
57
+
## Scaling (future)
63
58
64
-
## Scaling
65
-
66
-
Horizontal scaling is handled by K8s HPA. Each service exposes a `/metrics` endpoint (Prometheus format):
59
+
The current deployment is single-process. The architecture is designed so that services *could* be separated and scaled independently in the future. Potential scale-out dimensions:
Streamer is **not** proxied through Gateway. Video streaming data should not pass through a reverse proxy — clients access Streamer directly. This is by design: Gateway handles control-plane traffic only.
30
31
31
32
Adding endpoints to any downstream service requires **zero Gateway changes**.
-**AuthMiddleware** — extracts user identity and injects `X-User-Id` header for downstream services. Currently hardcoded to `bot` (single-user mode). Interface is defined so a real implementation (e.g. JWT verification) can be swapped in without changing Gateway internals or any downstream service.
40
41
-**RateLimitMiddleware** — per-user throttling using a backing cache (Redis / in-process). Only applies to control-plane requests since data-plane traffic never hits Gateway.
41
42
42
-
## Service Discovery on K8s
43
+
## Service Discovery
43
44
44
-
No external service registry needed. Gateway resolves upstreams via **K8s Service DNS**:
45
+
In the current single-process deployment, Gateway forwards requests to MetaService and Ingestor in-process. Upstream addresses are configured via environment variables:
45
46
46
47
```yaml
47
48
upstreams:
48
-
meta: "http://meta-service:8080"
49
-
ingest: "http://ingest-service:8080"
50
-
stream: "http://stream-service:8080"
49
+
meta: "http://localhost:8080"
50
+
ingest: "http://localhost:8080"
51
51
```
52
52
53
-
K8s handles DNS resolution and load balancing across pods.
54
-
55
53
## Why a Custom Gateway (Not Nginx)
56
54
57
55
K8s Ingress (Nginx/Envoy) handles infrastructure concerns: TLS termination, external routing, health checks. Gateway handles **business concerns**: auth validation and rate limiting are application logic that belongs in application code, not Nginx config.
1.**Single binary** — `app` is the only crate with `main.rs`. It parses CLI args to decide which service(s) to start. No other crate produces a binary.
20
20
2.**Library-first** — Every service crate is a lib crate that exposes its `Router` and `State`. The `app` crate composes them.
21
21
3.**`core` is shared code** — Domain types, database repositories, storage helpers, config. No HTTP server logic.
22
-
4.**`server` is the HTTP foundation** — Wraps axum with cross-cutting concerns: metrics endpoint for k8s HPA, health checks, graceful shutdown, tracing setup. Service crates depend on `server`, not on raw axum server setup.
22
+
4.**`server` is the HTTP foundation** — Wraps axum with cross-cutting concerns: health checks, graceful shutdown, tracing setup. Service crates depend on `server`, not on raw axum server setup.
23
23
5.**Each crate owns its tests** — Every lib crate has its own unit/integration tests. `cargo test -p <crate>` must pass independently.
Copy file name to clipboardExpand all lines: docs/design/meta-service.md
+5-31Lines changed: 5 additions & 31 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,8 +8,8 @@ Four responsibilities:
8
8
9
9
1.**Metadata management** — video CRUD, share link lifecycle, ownership enforcement
10
10
2.**Connection token** — issues HMAC-signed tokens that authorize clients to access data-plane services
11
-
3.**Topology awareness** — watches K8s Endpoints API to maintain a live service map
12
-
4.**Task lifecycle** — publishes cancellation/deletion events to the queue for Ingestor to consume
11
+
3.**Service address** — provides the service address for clients to connect to data-plane services
12
+
4.**Task lifecycle** — publishes cancellation/deletion events to the in-memory queue for Ingestor to consume (same process)
13
13
14
14
All mutating operations check ownership via `X-User-Id` (currently hardcoded to `bot` by Gateway). Each video record stores an `owner` field set at creation time.
15
15
@@ -19,20 +19,11 @@ All mutating operations check ownership via `X-User-Id` (currently hardcoded to
Follows the **Kafka / Redis Cluster pattern**: routing decisions live on the client side, MetaService only provides the data.
23
-
24
22
When a client requests an upload or playback, MetaService returns:
25
23
26
-
-**Service map** — list of healthy Ingestor/Streamer external addresses (NodePort)
24
+
-**Service address** — the address of the single-process server hosting Ingestor/Streamer
27
25
-**Connection token** — authorizes the client to connect
28
26
29
-
The client computes the target locally:
30
-
31
-
| Action | Strategy | Reason |
32
-
|--------|----------|--------|
33
-
| Upload | Round-robin | Stateless, no affinity needed |
34
-
| Playback | Consistent hash ring on `video_id`| Cache affinity — same video hits same Streamer. On node changes, only `1/N` keys re-map |
35
-
36
27
## Connection Token
37
28
38
29
Data-plane services (Ingestor/Streamer) sit outside Gateway's auth middleware. The connection token is the **only** mechanism that authorizes data-plane access. Without a valid token, services reject the connection.
`server_secret` is shared between MetaService and data-plane services via K8s Secret. Data-plane services verify tokens locally — no callback to MetaService needed.
39
+
`server_secret` is shared between MetaService and data-plane services via configuration. In the single-process deployment, all services share the same config. Data-plane services verify tokens locally — no callback to MetaService needed.
49
40
50
41
### Fields
51
42
@@ -67,23 +58,6 @@ The token does not encode a target instance. It only authorizes the action — t
-**Transmux processing** (CPU bound): ~5 concurrent per instance
78
80
79
-
With 3 instances × 5 transmux slots = ~15 videos/min throughput. HPA scales on queue depth; the queue provides natural backpressure (client sees `processing` status until complete).
81
+
In the single-process deployment, transmux concurrency is limited by available CPU cores. The in-memory queue provides natural backpressure (client sees `processing` status until complete).
80
82
81
83
### Streamer — Network I/O
82
84
@@ -88,13 +90,11 @@ With 3 instances × 5 transmux slots = ~15 videos/min throughput. HPA scales on
88
90
89
91
### Consistent hash rebalance on scale-up
90
92
91
-
Adding a Streamer instance remaps `1/N` of cached videos, causing cold-start penalties. Mitigate by scaling during low-traffic periods and using virtual nodes to reduce per-event disruption.
92
-
93
93
### Infrastructure
94
94
95
-
-**MinIO** — disk I/O on write path; use erasure coding across nodes + SSD for hot tier
96
-
-**Database** — ~10K writes/s (Postgres), sufficient for this scale; shard by `video_id` if needed
97
-
-**Queue** — ~100K msg/s (Redis Streams), far exceeds needs
95
+
-**MinIO** — disk I/O on write path; use SSD for hot tier
96
+
-**Database** — ~10K writes/s (Postgres), sufficient for this scale
0 commit comments