Skip to content

Commit e3ec050

Browse files
committed
docs: distinguish QMCB vs pipeline ColumnarBatch types
Add section explaining the two distinct ColumnarBatch interfaces: - QMCB (columnar.ts): ArrayBuffer-backed, wire format for DO transfer - Pipeline (operators.ts): Map<string, DecodedValue[]> with selection vector, used inside operators implementing nextColumnar()
1 parent 66e02d8 commit e3ec050

1 file changed

Lines changed: 15 additions & 0 deletions

File tree

docs/src/content/docs/columnar-format.mdx

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,3 +115,18 @@ Worker exit guard:
115115
```
116116

117117
WASM never calls back into JS. JS→WASM call overhead is 6ns cold, 0ns after TurboFan inlines (~1M calls). Shared `wasm.memory` access from JS is identical cost to regular typed arrays.
118+
119+
## Two ColumnarBatch types
120+
121+
The codebase has two distinct `ColumnarBatch` interfaces for different layers:
122+
123+
| Type | Module | Shape | Purpose |
124+
|------|--------|-------|---------|
125+
| **QMCB ColumnarBatch** | `columnar.ts` | `columns: ColumnarColumn[]` (ArrayBuffer-backed) | Wire format for DO-to-DO transfer |
126+
| **Pipeline ColumnarBatch** | `operators.ts` | `columns: Map<string, DecodedValue[]>`, `selection?: Uint32Array` | Operator-internal data flow |
127+
128+
The **QMCB** version is what this page documents — serialized to binary, transferred over RPC, decoded at the receiver. It uses typed array views (`Float64Array`, `BigInt64Array`) into raw `ArrayBuffer` data.
129+
130+
The **pipeline** version lives inside operators implementing `nextColumnar()`. It uses `Map<string, DecodedValue[]>` for decoded column values and an optional `selection` vector (a `Uint32Array` of active row indices) to mark post-filter survivors without copying column data. When a `ScanOperator` applies WASM SIMD filters, the result is a pipeline `ColumnarBatch` with a selection vector — only selected rows proceed to the next operator.
131+
132+
If you're writing a **custom operator** with `nextColumnar()`, you work with the pipeline type. If you're working with **inter-DO transfer**, you work with the QMCB type.

0 commit comments

Comments
 (0)