Skip to content

Commit 8a19ed9

Browse files
committed
docs: expand vector search — HNSW index building, serialization, tuning, IVF-PQ comparison
1 parent 41aaa0c commit 8a19ed9

File tree

1 file changed

+62
-0
lines changed

1 file changed

+62
-0
lines changed

docs/src/content/docs/vector-search.mdx

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,68 @@ For larger datasets, create an IVF-PQ (Inverted File with Product Quantization)
6969

7070
IVF-PQ indexes are stored alongside data in R2 and loaded on first query.
7171

72+
### HNSW
73+
74+
For datasets where you need fast approximate nearest neighbor search with high recall, build an HNSW (Hierarchical Navigable Small World) index:
75+
76+
```typescript
77+
import { HnswIndex } from "querymode"
78+
79+
// Build index
80+
const index = new HnswIndex({
81+
dim: 128,
82+
metric: "cosine",
83+
M: 16, // max connections per node (default: 16)
84+
efConstruction: 200, // construction beam width (default: 200)
85+
})
86+
87+
// Add vectors one at a time
88+
index.add(0, vec0)
89+
index.add(1, vec1)
90+
91+
// Or batch add from a contiguous Float32Array
92+
const allVectors = new Float32Array(1000 * 128) // 1000 vectors, 128 dims
93+
index.addBatch(allVectors, 128)
94+
95+
// Search
96+
const { indices, scores } = index.search(queryVec, 10, /* efSearch */ 50)
97+
// indices: Uint32Array of nearest neighbor IDs
98+
// scores: Float32Array of distances (lower = more similar)
99+
```
100+
101+
#### HNSW tuning
102+
103+
| Parameter | Default | Effect |
104+
|-----------|---------|--------|
105+
| `M` | 16 | Higher = better recall, more memory. 12-48 typical. |
106+
| `efConstruction` | 200 | Higher = better index quality, slower build. 100-400 typical. |
107+
| `efSearch` | topK | Higher = better recall at query time, slower search. Set to 2-4x topK. |
108+
109+
#### Serialization
110+
111+
HNSW indexes can be serialized to binary for storage (R2, disk) and deserialized on load:
112+
113+
```typescript
114+
// Save
115+
const binary: ArrayBuffer = index.serialize()
116+
await bucket.put("indexes/embeddings.hnsw", binary)
117+
118+
// Load
119+
const data = await bucket.get("indexes/embeddings.hnsw")
120+
const restored = HnswIndex.deserialize(await data.arrayBuffer(), "cosine")
121+
const results = restored.search(queryVec, 10)
122+
```
123+
124+
### IVF-PQ vs HNSW
125+
126+
| | IVF-PQ | HNSW |
127+
|--|--------|------|
128+
| **Speed** | Fast (quantized distances) | Fast (graph traversal) |
129+
| **Memory** | Low (compressed codes) | High (full vectors + graph) |
130+
| **Recall** | Good with enough probes | Excellent |
131+
| **Build time** | Requires training (k-means) | Incremental (add one at a time) |
132+
| **Best for** | Large datasets (>1M vectors) | Medium datasets (<1M vectors) |
133+
72134
## Combining with filters
73135

74136
Vector search composes with all other DataFrame operations:

0 commit comments

Comments
 (0)