diff --git a/docs/data-ingestion/architecture-diagram.jpg b/docs/data-ingestion/architecture-diagram.jpg new file mode 100644 index 00000000..0f03e20c Binary files /dev/null and b/docs/data-ingestion/architecture-diagram.jpg differ diff --git a/docs/data-ingestion/architecture-fit-table.png b/docs/data-ingestion/architecture-fit-table.png new file mode 100644 index 00000000..36359767 Binary files /dev/null and b/docs/data-ingestion/architecture-fit-table.png differ diff --git a/docs/data-ingestion/source-summary-table.png b/docs/data-ingestion/source-summary-table.png new file mode 100644 index 00000000..f652fa74 Binary files /dev/null and b/docs/data-ingestion/source-summary-table.png differ diff --git a/docs/images/architecture_diagram.jpg b/docs/images/architecture_diagram.jpg new file mode 100644 index 00000000..0f03e20c Binary files /dev/null and b/docs/images/architecture_diagram.jpg differ diff --git a/docs/images/architecture_fit_mapping.png b/docs/images/architecture_fit_mapping.png new file mode 100644 index 00000000..f652fa74 Binary files /dev/null and b/docs/images/architecture_fit_mapping.png differ diff --git a/docs/images/data_source_overview.png b/docs/images/data_source_overview.png new file mode 100644 index 00000000..36359767 Binary files /dev/null and b/docs/images/data_source_overview.png differ diff --git a/docs/images/unified_ingestion_arch_v2.jpg b/docs/images/unified_ingestion_arch_v2.jpg new file mode 100644 index 00000000..0f03e20c Binary files /dev/null and b/docs/images/unified_ingestion_arch_v2.jpg differ diff --git a/docs/unified_ingestion_architecture.md b/docs/unified_ingestion_architecture.md new file mode 100644 index 00000000..a70a44c5 --- /dev/null +++ b/docs/unified_ingestion_architecture.md @@ -0,0 +1,16 @@ +# Unified Ingestion Architecture (V2) + +This updated diagram provides a clearer view of both **real-time** and **batch** data ingestion pipelines. + +![Unified Ingestion Architecture](./images/unified_ingestion_arch_v2.jpg) + +## Key Components + +1. **WebSockets** – for real-time data flow to users. +2. **Redis Pub/Sub** – collects real-time data from sources and routes them to appropriate servers. +3. **Kafka Topics** – ensures high-throughput and avoids real-time data loss. +4. **Redis Insight** – for monitoring data flows. +5. **Redis Streams** – buffers batch data ingestion from various APIs, mobile apps, etc., ensuring durability and backpressure handling. + +Both systems ultimately write to a **PostgreSQL database** through worker consumers. +