feat: multi-tenant gRPC storage with embedded Rust in NativeAOT

[valdo404]

Summary

Complete multi-tenant architecture with gRPC storage, SSE proxy with PAT auth, Google Drive integration, and cloud source support — replacing direct filesystem access with a proper storage abstraction layer.

Architecture

• 3 gRPC services (proto/storage.proto): StorageService, SourceSyncService, ExternalWatchService
• Dual-server storage split: IHistoryStorage (sessions/WAL/index/checkpoints — can be remote R2) + ISyncStorage (file sync/watch — always local embedded or GDrive)
• Caller-orchestrated auto-save: Tool classes call sync.MaybeAutoSave() after sessions.AppendWal(). SessionManager and SyncManager are fully independent.
• Rust storage server (docx-storage-local): local filesystem backend with tenant-aware paths, file locking, SHA256-based change detection
• Embedded mode (new): Rust staticlib linked directly into .NET NativeAOT binaries — single binary, no separate process needed. In-memory DuplexStream transport for gRPC.
• Remote mode: standalone docx-storage-local binary over TCP/Unix socket for server deployments
• Cloudflare backend (docx-storage-cloudflare): R2-only storage with ETag-based optimistic locking (CAS), no KV dependency

Key changes

• IStorageClient split: monolithic IStorageClient/StorageClient → IHistoryStorage/HistoryStorageClient + ISyncStorage/SyncStorageClient
• SessionManager refactored: depends only on IHistoryStorage (removed all sync/watch/tracker logic)
• SyncManager created: depends only on ISyncStorage, handles RegisterAndWatch, MaybeAutoSave, Save, StopWatch
• 13 mutation call sites updated: PatchTool, StyleTools (×3), CommentTools (×2), RevisionTools (×3), HistoryTools (×3), ElementTools pass-through
• Embedded Rust FFI: lib.rs exposes C entry points (docx_storage_init, docx_pipe_read/write/flush, docx_storage_shutdown), .NET calls via P/Invoke (NativeStorage.cs, InMemoryPipeStream.cs)
• External change flow: Rust detects file changes (SHA256), streams events to .NET, .NET creates WAL entries (requires Open XML SDK for DocumentSnapshot)
• Rust register_source fix: creates SessionIndexEntry if absent (dual-server mode: AddSessionToIndex goes to remote, but RegisterSource goes to local)
• Format compatibility: 8-byte .NET header on DOCX/checkpoints preserved
• publish.sh: builds Rust staticlib → links into NativeAOT binaries + optional standalone binary
• Docker: multi-stage build (Rust staticlib → .NET NativeAOT), single binary runtime with build cache
• Infra: Pulumi IaC for Cloudflare resources, CI path filters, macOS/Windows builds restricted to workflow_dispatch

Phase H — SSE Proxy multi-tenant ✅

• HTTP reverse proxy (Rust/Axum) with PAT auth via Cloudflare D1
• Transparent SSE/JSON forwarding with X-Tenant-Id injection
• SessionManagerPool for multi-tenant single-process MCP
• Session recovery: proxy detects 404 (backend restart), transparently re-initializes MCP session and retries — per-tenant recovery lock prevents duplicate init under concurrent load
• Docker Compose: proxy → mcp-http → storage (local) or proxy → mcp-http → storage + gdrive (cloud)

Phase G — Google Drive gRPC server ✅

• docx-storage-gdrive: SourceSyncService + ExternalWatchService for Google Drive
• Per-tenant OAuth tokens from D1 (oauth_connection table), auto-refresh via TokenManager
• URI format: gdrive://{connection_id}/{file_id}
• File download, upload (update existing), and create new file on first sync
• Website OAuth consent flow (connect, callback, connections API)
• ConnectionsManager component in dashboard for browsing external storage

Cloud Source Support ✅

• DocumentTools.DocumentOpen supports cloud sources (Google Drive) via source_type + connection_id + file_id
• ResolveSourceType() infers source type and blocks local sources in cloud mode
• SyncManager.ReadSourceBytes() abstracts local vs cloud file reads
• ExternalChangeGate + ExternalChangeTools work with cloud sources (not just local disk)
• DocumentSave preserves existing source type on save-as
• ExternalChangeGate state persisted in gRPC storage index (survives backend restart)

Website ✅

• Astro 6 + Cloudflare Pages + Better Auth + D1
• OAuth connections management (Google Drive)
• Dashboard with ConnectionsManager for browsing cloud storage
• Mobile responsive layout
• i18n (FR/EN)

Pending

Phase D — Validation

• Large document validation (>50MB) — streaming checkpoints/WAL

Phase W — WAL/Sessions viewer

• HTTP server for visualizing sessions and WAL entries stored on R2

Phase K — Déploiement Koyeb

• Deploy SSE proxy, .NET MCP server, Cloudflare storage on Koyeb

Infra

• Cross-platform Docker builds (linux-x64, linux-arm64)

Relates to

• provide online sse integration that is compatible with microsoft 365 #20 (provide online SSE integration) — SSE proxy + R2 storage + GDrive sync done
• Bug: mcp client should know when a new entry has been inserted #44 (mcp client notification on new entry) — ExternalChangeGate + acknowledgment flow implemented
• Feat: build a get diff tool. #46 (get diff tool) — not yet started

Test plan

• 428 .NET unit tests passing (embedded mode)
• 428 .NET unit tests passing (dual-server mode with STORAGE_GRPC_URL → Cloudflare R2)
• 0 skipped tests — all auto-save, external sync, and external change tracker tests run in both modes
• 31 Rust unit tests passing
• 63 MCP integration tests passing (via mcptools)
• CLI lifecycle test (open, add, query, replace-text, undo, redo, history, export, save, close)
• NativeAOT publish with embedded Rust (35MB single binary)
• Cloudflare R2 storage backend (ETag CAS, no KV dependency)
• Dual-server TestHelpers (IHistoryStorage→remote, ISyncStorage→local embedded)
• Docker Compose full stack (storage + gdrive + mcp-http + proxy) — all 4 services healthy
• Session recovery verified (proxy detects 404 on backend restart, re-initializes transparently)
• Cross-platform Docker builds (linux-x64, linux-arm64)
• Large document stress tests (>50MB)

🤖 Generated with Claude Code

[valdo404]

missing:

the whole is working into docker compose
the whole is working into koyeb

it integrates into claude desktop / claude.ai