You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: add nextColumnar() implementation guide for custom operators
New section in operators.mdx with working example (DoublingOperator),
explains pipeline ColumnarBatch shape (Map + selection vector), links
to columnar-format page for QMCB distinction, and guidance on when
to implement vs skip.
Copy file name to clipboardExpand all lines: docs/src/content/docs/operators.mdx
+42Lines changed: 42 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -108,6 +108,48 @@ For numeric columns, the `WasmAggregateOperator` uses Zig SIMD vector instructio
108
108
109
109
The DataFrame API automatically selects WASM aggregates when available.
110
110
111
+
## Columnar fast path (nextColumnar)
112
+
113
+
Operators can optionally implement `nextColumnar()` alongside `next()` for zero-copy performance. When the pipeline detects a columnar-capable operator chain, it uses `nextColumnar()` to pass column data directly without creating `Row[]` objects.
The pipeline `ColumnarBatch` (from `operators.ts`) uses `Map<string, DecodedValue[]>` for columns and an optional `selection?: Uint32Array` for post-filter row indices. This is distinct from the QMCB wire-format `ColumnarBatch` documented in [Columnar Format](/columnar-format/) — see that page for the distinction.
148
+
149
+
When to implement `nextColumnar()`:
150
+
-**Do**: numeric transforms on large datasets (avoids `Row[]` allocation)
151
+
-**Skip**: operators that reshape data (joins, aggregates) or need random access across rows
152
+
111
153
## Why composable operators matter
112
154
113
155
Traditional engines give you a fixed query language. You can't put a window function before a join, run custom logic between pipeline stages, or swap the sort implementation.
0 commit comments