feat: add hnsw-rabitq support

[egolearner]

resolve #42

Greptile Summary

This PR introduces a new HNSW-RaBitQ index type (HNSW_RABITQ) to zvec, combining the HNSW graph structure with RaBitQ scalar quantization for approximate nearest-neighbor search. The implementation spans a new hnsw_rabitq algorithm directory (~7000 lines), full builder/searcher/streamer support, Python bindings, DB-layer integration, and tests.

Key changes:

• New RabitqConverter (K-means clustering + rotation) and RabitqReformer (quantization + centroid look-up) components implement the two-phase offline/online quantization pipeline.
• HnswRabitqBuilder performs a two-phase build: first quantizes all vectors via RabitqReformer::convert, then builds the HNSW graph in parallel.
• HnswRabitqSearcher and HnswRabitqStreamer support all three search modes (standard, brute-force, and p-keys), including group-by and filter operations.
• DB schema validation for HNSW_RABITQ correctly gates on platform (Linux x86_64), CPU capabilities (AVX2/AVX512F), and data type (FP32).

Issues found:

• The stale error message for unsupported dense-vector index types (in schema.cc) lists only FLAT|HNSW|IVF and does not mention HNSW_RABITQ, which was added to the supported set in this PR.
• In hnsw_rabitq_query_algorithm.cc, after get_full_est is computed, nodes are unconditionally added to the candidates heap without re-checking the full estimate against the topk threshold, leading to unnecessary heap operations on the hot search path.
• Several issues identified in earlier review rounds (VLAs, null dereferences, POSIX-only headers, search_bf_by_p_keys_impl being a no-op, cluster->mount() return value ignored) have been addressed. The train(IndexTrainer::Pointer) overload leaving reformer_ null (and thus causing a crash in build()) remains unresolved per prior thread history.

Confidence Score: 3/5

• Not yet safe to merge — the train(IndexTrainer::Pointer) path leaves reformer_ null and will crash on any subsequent build() call; this should be resolved before merging.
• The core algorithm and search/stream paths are generally well-structured and most previously-flagged bugs have been fixed. However, the unresolved train(IndexTrainer::Pointer)→build() null dereference (tracked in prior review threads with no "fixed" reply) represents a crash-level regression on a documented public API path. The remaining open issues (unused <execinfo.h> include, stale error message, minor candidates-heap performance) are lower severity.
• Pay close attention to src/core/algorithm/hnsw_rabitq/hnsw_rabitq_builder.cc — specifically the train(const IndexTrainer::Pointer &) overload (lines 333–348) which does not initialize reformer_, and the subsequent build() call at line 401 that dereferences it.

Important Files Changed

Filename	Overview
src/core/algorithm/hnsw_rabitq/hnsw_rabitq_builder.cc	Core builder for HNSW-RaBitQ. Previous issues (VLA, null alg_, dead null-check, rand() thread safety) have been addressed. The train(IndexTrainer::Pointer) path still leaves reformer_ null, which causes a crash in build() at the reformer_->convert() call — flagged in prior review threads.
src/core/algorithm/hnsw_rabitq/hnsw_rabitq_query_algorithm.cc	Query-time HNSW search with RaBitQ distance estimation. VLA and ex_bits_==0 guard issues from prior reviews are fixed. A minor performance opportunity exists: after get_full_est, nodes are unconditionally added to candidates even when the full estimate is already worse than the topk boundary.
src/core/algorithm/hnsw_rabitq/rabitq_converter.cc	Handles K-means clustering and quantization training for RaBitQ. Previously flagged issues (mount() return code ignored, <strings.h> portability) are now fixed with proper error checking and #ifdef _MSC_VER guard.
src/core/algorithm/hnsw_rabitq/rabitq_reformer.cc	PIMPL-based reformer that loads centroids/rotator and transforms vectors/queries for RaBitQ. Load and convert paths are well-structured with proper error checking and memcpy-based data copying.
src/core/algorithm/hnsw_rabitq/hnsw_rabitq_searcher.cc	Searcher with brute-force, HNSW, and p_keys search paths. The previously commented-out search_bf_by_p_keys_impl body is now properly implemented. All three search modes are present and check context validity.
src/core/algorithm/hnsw_rabitq/hnsw_rabitq_streamer.cc	Real-time streaming index supporting concurrent add and search. Reformer initialization and dumping logic is correct. add_with_id_impl correctly serializes graph updates under a shared_mutex. An "orphaned vector" (vector added to entity but graph update fails) edge case exists but is common to all graph index streamers.
src/core/algorithm/hnsw_rabitq/hnsw_rabitq_entity.h	Core entity header defining node/key/level/distance types and graph data structures. The unused POSIX-only <execinfo.h> include (line 16) was flagged in prior review threads; no other structural issues.
src/db/index/common/schema.cc	Schema validation extended for HNSW_RABITQ: correct platform/CPU checks and dimension/metric/type constraints added. Minor stale error message: the unsupported-index-type diagnostic still lists only FLAT
src/core/algorithm/hnsw_rabitq/hnsw_rabitq_algorithm.cc	Build-time graph construction (add_node, select_entry_point, update_neighbors). VLA issues from prior reviews are fixed; std::vector is used throughout. Mutex locking protocol for concurrent max-level updates appears correct.

Sequence Diagram

sequenceDiagram
    participant Caller
    participant HnswRabitqBuilder
    participant RabitqConverter
    participant RabitqReformer
    participant HnswRabitqEntity
    participant HnswRabitqAlgorithm

    Caller->>HnswRabitqBuilder: init(meta, params)
    HnswRabitqBuilder->>RabitqConverter: init(meta, params)
    HnswRabitqBuilder->>HnswRabitqEntity: init()
    HnswRabitqBuilder->>HnswRabitqAlgorithm: init()

    Caller->>HnswRabitqBuilder: train(holder)
    HnswRabitqBuilder->>RabitqConverter: train(holder) [K-means clustering]
    HnswRabitqBuilder->>RabitqConverter: dump(MemoryDumper)
    HnswRabitqBuilder->>RabitqReformer: init + load(MemoryReadStorage)
    Note over HnswRabitqBuilder: reformer_ now ready

    Caller->>HnswRabitqBuilder: build(threads, holder)
    loop For each vector
        HnswRabitqBuilder->>RabitqReformer: convert(vec) → quantized_vec
        HnswRabitqBuilder->>HnswRabitqEntity: add_vector(level, key, quantized_vec)
    end
    loop Parallel threads
        HnswRabitqBuilder->>HnswRabitqAlgorithm: add_node(id, level, ctx)
    end

    Caller->>HnswRabitqBuilder: dump(dumper)
    HnswRabitqBuilder->>RabitqConverter: dump(dumper) [centroids + rotator]
    HnswRabitqBuilder->>HnswRabitqEntity: dump(dumper) [graph data]

    Note over Caller: Search path
    Caller->>HnswRabitqSearcher: load(storage)
    HnswRabitqSearcher->>RabitqReformer: load(storage)
    HnswRabitqSearcher->>HnswRabitqEntity: load(storage)
    Caller->>HnswRabitqSearcher: search(query, count, ctx)
    HnswRabitqSearcher->>RabitqReformer: transform_to_entity(query) → QueryEntity
    HnswRabitqSearcher->>HnswRabitqQueryAlgorithm: search(QueryEntity, ctx)

Loading

_{Last reviewed commit: "Merge remote-trackin..."}

[egolearner]

Depends on VectorDB-NTU/RaBitQ-Library#36

[egolearner]

@Greptile

[egolearner]

@greptile

[egolearner]

@greptile

[egolearner]

@greptile

[egolearner]

@greptile