feat: Add Unified Ingestion Architecture Diagram and Flow Design (#99)

[23shivay]

Summary

This PR addresses Task #99: Design Architecture for Unified, Fault-Tolerant Data Ingestion Layer by providing a detailed high-level architecture for real-time and batch ingestion. The submission includes an updated architecture diagram and accompanying notes that outline the ingestion flow, core components, and technology choices.

---

Key Additions

• Architecture Diagram
  Added unified_ingestion_arch_v2.jpg to docs/images/, capturing the complete ingestion pipeline.

• Documentation
  Added unified_ingestion_architecture.md to docs/, explaining the components, flow, and design rationale.

---

Highlights

• Real-Time Ingestion

• Redis Pub/Sub layer supports real-time event flow via WebSockets.

• Kafka used for high-throughput, durable message queuing.

• Batch Ingestion

• Redis Streams + worker consumers pull data from APIs, mobile apps, and web portals.

• Supports scheduled batch jobs and bulk payload processing.

• Storage

• PostgreSQL as the persistent, relational data store.

---

Design Goals Addressed

• Fault tolerance via decoupled queue-based architecture.
• Scalability through modular components and distributed message brokers.
• Extensibility with pluggable consumer/producer logic.
• Technology-agnostic design adaptable beyond Kafka.

---

cc @mvadodariya