You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: trim README from 351 to 89 lines — link to docs instead of duplicating
Kept: quickstart, pitch, build commands, status.
Removed: operator table, architecture diagram, full API reference, examples,
MapReduce explanation — all live in the docs site.
> **Experimental** — early prototype, not production-ready. Architecture and API will change.
4
4
5
+
A pluggable columnar query library — not a query engine you push data to, but a query capability your code uses directly. No data materialization, no engine boundary, no SQL transpilation.
// Or query your own files — Parquet, Lance, CSV, JSON, Arrow
26
+
// Query your own files — Parquet, Lance, CSV, JSON
27
27
const qm =QueryMode.local()
28
28
const result =awaitqm
29
29
.table("./data/events.parquet")
30
30
.filter("status", "eq", "active")
31
-
.filter("amount", "gte", 100)
32
-
.filter("amount", "lte", 500)
33
31
.select("id", "amount", "region")
34
32
.sort("amount", "desc")
35
33
.limit(20)
36
34
.collect()
37
-
```
38
-
39
-
A pluggable columnar query library — not a query engine you push data to, but a query capability your code uses directly. No data materialization, no engine boundary, no SQL transpilation.
40
-
41
-
**[Why QueryMode?](https://teamchong.github.io/querymode/why-querymode/)** — Agents need dynamic pipelines, not pre-built ETL. QueryMode lets the agent define both query and business logic in the same code, at query time, with no serialization boundary between stages.
42
-
43
-
## Why "mode" not "engine"
44
-
45
-
Every query engine — Spark, DataFusion, DuckDB, Polars — has a boundary between your code and the engine:
46
-
47
-
```
48
-
Traditional engine:
49
-
50
-
Your Code Engine
51
-
───────── ──────
52
-
filter(age > 25) ──────► translate to internal plan
53
-
materialize data into Arrow/DataFrame
54
-
run engine's fixed operators
55
-
serialize results
56
-
◄────── return results to your code
57
-
58
-
Your code CANNOT cross the boundary.
59
-
Custom business logic? Pull data out, process in your code, push back in.
60
-
That round-trip IS data materialization.
61
-
```
62
-
63
-
LINQ and ORMs look like code-first but they're transpiling expressions to SQL strings sent to a separate database. The database still materializes your data into its format, runs its fixed operators, and sends results back.
64
-
65
-
QueryMode has no boundary:
66
-
67
-
```typescript
68
-
// This IS the execution — not a description translated to SQL
// But orchestration is YOUR code, not a query planner's fixed operators
76
-
```
77
-
78
-
Your app code IS the query execution. The WASM engine is a library function your code calls — column decoding, SIMD filtering, vector search happen in-process, on raw bytes, zero-copy. There's no "register UDF → engine materializes data → calls your function → collects results" boundary.
79
-
80
-
**What this means in practice:**
81
-
- No data materialization — data stays in R2/disk, only the exact matching bytes are read
82
-
- No engine boundary — your business logic runs directly, not as a registered UDF
83
-
- No SQL transpilation — the API calls ARE the execution, not a description sent elsewhere
84
-
- No fixed operator set — your code can do anything between query steps
85
-
- Same binary everywhere — browser, Node/Bun, Cloudflare DO
86
-
87
-
## Query engine as code
88
-
89
-
Every query operation is a composable code primitive. They all implement the same pull-based `Operator` interface — `next() → RowBatch | null` — so you chain them however you want.
// Pull results — zero-copy, no serialization between stages
162
-
const rows =awaitdrainPipeline(top10)
163
-
```
164
-
165
-
### Or use the DataFrame API
166
-
167
-
The same operators power the fluent API — `.filter()` becomes `FilterOperator`, `.sort()` becomes `ExternalSortOperator`, etc:
168
-
169
-
```typescript
170
-
const qm =QueryMode.local()
171
-
const results =awaitqm
172
-
.table("orders")
173
-
.filter("amount", "gt", 100)
174
-
.groupBy("region")
175
-
.aggregate("sum", "amount", "total")
176
-
.sort("total", "desc")
177
-
.limit(10)
178
-
.exec()
179
-
```
180
-
181
-
Both paths produce the same pull-based pipeline. The DataFrame API is sugar; the operators are the engine.
182
-
183
-
### Memory-bounded with R2 spill
184
-
185
-
Operators that accumulate state (sort, join, aggregate) accept a memory budget. When exceeded, they spill to R2 via `SpillBackend` — same interface whether running on Cloudflare edge or local disk:
SQL is another way in — same operator pipeline underneath:
200
-
201
-
```typescript
202
-
const qm =QueryMode.local()
203
-
const results =awaitqm
204
-
.sql("SELECT region, SUM(amount) AS total FROM orders WHERE status = 'active' GROUP BY region ORDER BY total DESC LIMIT 10")
205
-
.collect()
206
35
207
-
// SQL and DataFrame compose — chain further operations after SQL
208
-
const filtered =awaitqm
209
-
.sql("SELECT * FROM events WHERE created_at > '2026-01-01'")
210
-
.filter("country", "eq", "US")
211
-
.sort("amount", "desc")
212
-
.limit(50)
36
+
// SQL works too — same operator pipeline underneath
37
+
const sql =awaitqm
38
+
.sql("SELECT region, SUM(amount) AS total FROM orders GROUP BY region ORDER BY total DESC")
213
39
.collect()
214
-
```
215
-
216
-
Supports: SELECT, WHERE (AND/OR/NOT, LIKE, NOT LIKE, IN, NOT IN, BETWEEN, NOT BETWEEN, IS NULL, IS NOT NULL), GROUP BY, HAVING, ORDER BY (multi-column), LIMIT/OFFSET, DISTINCT, CASE/CAST, arithmetic expressions, JOINs, window functions (ROW_NUMBER, RANK, LAG, LEAD), UNION/INTERSECT/EXCEPT.
217
-
218
-
### Why this matters
219
40
220
-
Traditional engines give you a fixed query language. You can't put a window function before a join, run custom logic between pipeline stages, or swap the sort implementation. The planner decides.
221
-
222
-
With QueryMode, operators are building blocks. Your code assembles the pipeline, controls the memory budget, decides when to spill. The query engine isn't a service you call — it's a library your code composes.
223
-
224
-
### Beyond traditional engines
225
-
226
-
These examples show what's possible when operators are composable building blocks, not a fixed plan:
227
-
228
-
| Example | What it shows | Why DuckDB/Polars can't |
|[`examples/ml-scoring-pipeline.ts`](examples/ml-scoring-pipeline.ts)| Custom scoring runs **inside** the pipeline between Filter and TopK | UDFs serialize data across the engine boundary |
231
-
|[`examples/adaptive-search.ts`](examples/adaptive-search.ts)| Vector search with adaptive threshold — recompose if too few results | Fixed query planner can't dynamically widen search |
232
-
|[`examples/custom-spill-backend.ts`](examples/custom-spill-backend.ts)| Pluggable spill storage (memory, R2, S3) at 4KB budget | DuckDB: disk only. Polars: no spill at all |
233
-
|[`examples/nextjs-api-route.ts`](examples/nextjs-api-route.ts)| Next.js/Vinext API route — query Parquet files, deploy to edge | DuckDB needs a sidecar process, can't run in Workers |
234
-
235
-
Run any example:
236
-
```bash
237
-
npx tsx examples/ml-scoring-pipeline.ts
238
-
npx tsx examples/adaptive-search.ts
239
-
npx tsx examples/custom-spill-backend.ts
240
-
npx tsx examples/nextjs-api-route.ts
41
+
// Edge mode — same API, WASM runs inside regional DOs
-**Zig WASM engine** (`wasm/`) — column decoding, SIMD ops, SQL execution, vector search, fragment writing, compiles to `querymode.wasm`
247
-
-**Code-first query API** — `.table().filter().select().sort().limit().exec()` or `.sql("SELECT ...")`, with `.toCode()` decompiler for logging and LLM context compression
248
-
-**Write path** — `append(rows, { path, metadata })` with CAS-based manifest coordination via Master DO, `dropTable()` for cleanup
249
-
-**Master/Query DO split** — single-writer Master broadcasts footer invalidations to per-region Query DOs
250
-
-**Footer caching** — table footers (~4KB each) cached in DO memory with VIP eviction (hot tables protected from eviction)
251
-
-**Bounded prefetch pipeline** — R2 range fetches overlap I/O (fetch page N+1 while WASM processes page N)
252
-
-**IVF-PQ vector search** — index-aware routing in Query DO, falls back to flat SIMD search when no index present
253
-
-**Multi-format support** — Lance, Parquet, and Iceberg tables
254
-
-**Local mode** — same API reads Lance/Parquet files from disk or HTTP (Node/Bun)
255
-
-**Fragment DO pool** — fan-out parallel scanning for multi-fragment datasets (one DO per fragment, scales with data)
256
-
-**600+ tests** — unit tests cover footer parsing, column decoding, Parquet/Thrift, merging, aggregates, VIP cache, WASM integration, SQL, partition catalog, materialized executor, toCode decompiler; 110+ conformance tests validate every operator against DuckDB at 1M-5M row scale
257
-
-**CI benchmarks** — head-to-head QueryMode (Miniflare) vs DuckDB (native) on every push, results posted to [GitHub Actions summary](https://github.com/teamchong/querymode/actions/workflows/ci.yml)
45
+
## What it is
258
46
259
-
## What doesn't exist yet
47
+
Operators are composable building blocks, not a fixed query plan. Your code assembles the pipeline, controls the memory budget, decides when to spill. The query engine isn't a service you call — it's a library your code composes.
260
48
261
-
- No deployed instance
262
-
- No browser mode
263
-
- No npm package published (install from source via git clone)
WASM is slower than native (~1.3–1.5× overhead), and a single Durable Object has hard memory and CPU caps. You can't build a competitive query engine by running everything in one WASM instance on one node.
71
+
## What exists
340
72
341
-
QueryMode doesn't try. It uses the network as a distributed compute fabric — like biological cells, not a brain:
73
+
- 600+ tests, 110+ conformance tests validated against DuckDB at 1M-5M row scale
74
+
- CI benchmarks: QueryMode (Miniflare) vs DuckDB (native) on every push
- Memory-bounded operators with R2 spill (sort, join, aggregate)
77
+
- IVF-PQ vector search with flat SIMD fallback
78
+
- Zero-copy columnar pipeline (QMCB binary format, no Row[] until response boundary)
79
+
- Local mode (Node/Bun) and edge mode (Cloudflare Workers)
342
80
343
-
-**DOs as cells** — every Fragment DO carries the same WASM binary (DNA). They activate on signal, scan their fragment, and go dormant. More data → more cells. Idle cells cost nothing (they hibernate).
344
-
-**R2 as virtual memory** — when a single DO's 128MB fills up, operators spill to R2. The pipeline doesn't care if data is in-memory or spilled — same interface, unbounded capacity.
345
-
-**Fan-out as bandwidth** — more fragments = more parallel R2 reads = more aggregate throughput. No cell coordinates with another — they all respond to the same signal independently.
81
+
## What doesn't exist yet
346
82
347
-
QueryDO **maps** fragments to Fragment DOs, each DO runs WASM SIMD on its shard, then QueryDO **reduces** via k-way merge. No single node does heavy work. The code is the DNA — scale comes from more cells, not smarter ones. See [Architecture](https://teamchong.github.io/querymode/architecture/) for the full deep dive.
0 commit comments