drs-backend is an experimental semantic search engine based on the Directed Recomputational Search (DRS) framework. Heavily inspired by the LEANN architecture, this Go implementation explores the radical tradeoff of exchanging storage for computation.
In traditional vector databases, the architecture acts as a "cartographer," attempting to store massive, dense mathematical representations of every data fragment. drs-backend follows the DRS paradigm: true intelligence resides not in storing the object, but in the mastery of reconstructing it exactly when needed.
- Ontological Separation: Strict distinction between Persistent Structural State () and Derivable Exact State ().
- Minimal Persistence: Zero storage of dense embeddings. We only persist the structural DNA (graph topology + quantization codes) needed to guide the search.
- Total Derivability: All exact signals are recomputed on-the-fly from raw data and the query, ensuring absolute semantic fidelity.
- Strict DRS Implementation: Not a "pure approximation" system; the final decision is always governed by exact signals.
- Off-Heap Memory Management: Optimized for Go using Argo, bypassing Garbage Collector (GC) overhead for massive structural indices.
- Recomputation as a Service: Explicit control over the recomputation budget to manage hardware utilization (GPU/Inference) vs. latency.
- Semantic Correction Proofs: Comprehensive test suites (including the "Approx Lies" test) ensuring that DRS corrects approximation noise.
- Query Encoding: The incoming query is encoded into an exact embedding (not persisted).
- Graph Navigation (Level 1): The space is explored using cheap, approximate signals (e.g., PQ codes) stored off-heap.
- Directed Recomputation (Level 2): Promising candidates identified during exploration are sent to the
EmbeddingServerManagerfor exact reconstruction. - Final Decision: The top-k ranking is based exclusively on exact recomputed distances.
drs-backend/
├── internal/
│ ├── searcher/ # Online DRS Algorithm & Two-level logic
│ │ ├── algorithm.go
│ │ ├── queues.go # Exploration vs. Decision structures
│ │ └── *_test.go
│ ├── graph/ # CSR Graph structures (Argo-managed off-heap)
│ ├── embedding/ # Recomputation clients (Real Inference + Mocks)
│ └── types/ # Shared architectural contracts
├── docs/ # Deep dives into the DRS Ontology
├── go.mod
└── README.md
Important
Licensing Information: This project is Source-Available. It is free for personal, academic, and research use. Commercial use (SaaS, internal production, or redistribution) requires a separate license. See ⚖️ Licensing for details.
This software and the specific implementation of the DRS paradigm herein are the proprietary technology of [Your Name/Company].
- Personal/Research Use: Free and encouraged.
- Commercial Use: Prohibited without a Commercial License Agreement. This includes SaaS deployments, internal production tools, or inclusion in proprietary products.
For commercial inquiries, please contact: [Your Email/Contact]
We welcome contributions to optimize this engine! Please note that by submitting a Pull Request, you agree to our Contributor License Agreement (CLA). By contributing, you grant the author the irrevocable right to use, modify, and re-license your contributions as part of both commercial and non-commercial versions of this project.
See CONTRIBUTING.md for the full CLA text.
- Correct and stable DRS search loop.
- Basic recomputation batching.
- Off-heap memory integration via Argo.
- Global batching & Inference caching.
- Graph pruning & HNSW/DiskANN backend integration.
- Real-time Product Quantization (PQ) scoring.
- High-performance Go SIMD distance kernels.
- Correctness First: Ensure the algorithm obeys the 14 DRS Invariants.
- Understanding over "Black Magic": Explicit logic over opaque optimizations.
- Performance Second: Optimization follows algorithmic verification.
If you understand this code, you understand the future of efficient RAG and high-scale vector retrieval.