Implement cost-based optimizer for INTERSECTS join algorithm selection — Closes #67

[conradbzura]

Summary

Add a DataFusion logical optimizer rule that rewrites INTERSECTS joins into binned equi-joins with adaptive bin sizing. Instead of reverse-engineering overlap predicates from column-name heuristics after transpilation, introduce a "datafusion" dialect to the GIQL transpiler that emits giql_intersects() function calls, preserving INTERSECTS semantics through the SQL layer so the optimizer can match directly.

Bin size is chosen using two-tier statistics: Tier 1 reads actual start/end values from 1–3 representative Parquet row groups and computes p95 interval width (~0.2ms overhead, capped at 300K sampled rows). Tier 2 falls back to column-level min/max estimates for non-Parquet sources. DataFusion handles parallelism, hash join execution, and disk spilling natively.

Closes #67

Proposed changes

Transpiler dialect support (Python)

Add dialect parameter to transpile() with "datafusion" option. A new DataFusionGIQLGenerator subclass overrides _generate_column_join() for INTERSECTS to emit giql_intersects(start, end, start, end) instead of raw overlap predicates. The chrom equi-key is preserved as standard SQL for hash partitioning. Literal range queries and CONTAINS/WITHIN fall through to the base generator unchanged.

Placeholder UDF (lib.rs)

Register a giql_intersects ScalarUDF on the DataFusion session state so the SQL parser accepts the function call. The UDF is never executed — the logical rule rewrites it away before execution. If the rule is missing, the UDF returns a clear error message.

Logical optimizer rule (logical_rule.rs)

The IntersectsLogicalRule detects giql_intersects() in join filters and rewrites to:

1. Expand both sides with range(start/B, (end-1)/B+1) bin columns via UNNEST
2. Equi-join on (chrom, bin_id) plus the original overlap filter (with giql_intersects() replaced by real start < end AND end > start predicates)
3. Canonical-bin dedup filter (__giql_bins == CASE WHEN left_first_bin >= right_first_bin THEN left_first_bin ELSE right_first_bin END) eliminates multi-bin duplicates without DISTINCT
4. Project away internal bin columns

Table aliases for the UNNEST SubqueryAlias are derived from the plan's schema qualifier rather than walking to the underlying TableScan, which correctly handles SQL aliases and self-joins.

Two-tier adaptive bin sizing (logical_rule.rs)

Tier 1 — Parquet sampling (~0.2ms): Read start/end columns from 1–3 representative row groups (first, middle, last) using the parquet crate's Arrow reader with projection pushdown. Compute actual interval widths (filtering non-positive values from malformed data) and return p95. Sampling is capped at 300K rows to bound memory for very large row groups. This handles all width distributions correctly, including "middle-wide" data where the widest interval has neither the smallest start nor the largest end. For remote sources (s3://, gs://), sampling fails gracefully and falls through to Tier 2.

Tier 2 — Column-level heuristic (fallback): For non-Parquet sources, estimate width from max(min(end)-min(start), max(end)-max(start)) — two independent estimates from column-level min/max stats, taking the max for robustness.

Both tiers clamp to [1,000, 1,000,000] and fall back to a default of 10,000 when no statistics are available.

Build configuration

Add Rust gitignore patterns (target/, Cargo.lock) to both the root .gitignore and a crate-level .gitignore. The parquet crate is a runtime dependency for Tier 1 sampling.

Test cases

#	Test Suite	Given	When	Then	Coverage Target
1	logical_rule_test	Default rule instance	Rule name queried	Returns intersects_logical_binned	Trait impl
2	logical_rule_test	Default rule instance	Apply order queried	Returns BottomUp	Trait impl
3	logical_rule_test	Default rule instance	supports_rewrite queried	Returns true	Trait impl
4	logical_rule_test	An EmptyRelation plan	Rule applied	Plan unchanged	Non-join skip
5	logical_rule_test	LEFT JOIN with giql_intersects	Rule applied	Plan not transformed	Join type filter
6	logical_rule_test	RIGHT JOIN with giql_intersects	Rule applied	Plan not transformed	Join type filter
7	logical_rule_test	FULL OUTER JOIN with giql_intersects	Rule applied	Plan not transformed	Join type filter
8	logical_rule_test	INNER JOIN with raw overlap predicates	Rule applied	Plan not transformed	Function detection
9	logical_rule_test	Three overlapping intervals across two tables	INTERSECTS join executed	Returns 3 pairs	Basic correctness
10	logical_rule_test	Adjacent intervals [100,200) and [200,300)	INTERSECTS join executed	Returns 0 pairs	Half-open semantics
11	logical_rule_test	Interval [100,500) containing [200,300)	INTERSECTS join executed	Returns 1 pair	Containment
12	logical_rule_test	Intervals on different chromosomes	INTERSECTS join executed	Returns 0 pairs	Chrom isolation
13	logical_rule_test	Wide intervals spanning multiple bins	Binned join executed	Each pair appears once	Canonical-bin dedup
14	logical_rule_test	One very wide interval vs multiple narrow	Binned join executed	Each pair appears once	Multi-bin dedup
15	logical_rule_test	100 narrow intervals (width ~100bp)	INTERSECTS join executed	Returns 100 pairs	Small bin sizing
16	logical_rule_test	Wide intervals (width ~50,000bp)	INTERSECTS join executed	Returns 2 pairs	Large bin sizing
17	logical_rule_test	Intervals on chr1, chr2, chr3 vs chr1, chr2, chr4	INTERSECTS join executed	Returns 2 pairs	Multi-chromosome
18	logical_rule_test	3x3 many-to-many overlapping intervals	INTERSECTS join executed	Returns 8 pairs	Combinatorial
19	logical_rule_test	Non-empty left, empty right table	INTERSECTS join executed	Returns 0 rows	Empty table
20	logical_rule_test	Two single-row overlapping tables	INTERSECTS join executed	Returns 1 pair	Minimal input
21	logical_rule_test	Two single-row non-overlapping tables	INTERSECTS join executed	Returns 0 pairs	Minimal input
22	logical_rule_test	Two tables with identical intervals	INTERSECTS join executed	Returns 4 pairs (2x2)	Duplicate intervals
23	logical_rule_test	Intervals overlapping by exactly 1bp	INTERSECTS join executed	Returns 1 pair	Boundary
24	logical_rule_test	Tables with chromStart/chromEnd columns	giql_intersects with custom names	Returns 2 pairs	Custom column names
25	logical_rule_test	Known overlapping intervals	INTERSECTS join executed	Output values match expected	Value correctness
26	logical_rule_test	Tables named peaks/genes	INTERSECTS join executed	Returns 3 pairs	Non-standard aliases
27	logical_rule_test	Single table joined against itself	INTERSECTS self-join executed	Returns 5 pairs	Self-join aliases
28	logical_rule_test	Extra filter alongside giql_intersects	INTERSECTS join with b.start > 300	Returns 1 pair	Compound predicates
29	logical_rule_test	Middle-wide distribution where widest interval has neither min(start) nor max(end)	INTERSECTS join executed	Returns 2 pairs with correct dedup	Parquet sampling robustness
30	lib::tests	Fresh SessionState	register_optimizer called	Rule and UDF both registered	Registration
31	TestTranspileDataFusionDialect	Column-to-column INTERSECTS join	Transpile with dialect="datafusion"	Output contains giql_intersects()	Dialect routing
32	TestTranspileDataFusionDialect	Literal range INTERSECTS	Transpile with dialect="datafusion"	Output uses standard SQL predicates	Literal fallback
33	TestTranspileDataFusionDialect	CONTAINS column join	Transpile with dialect="datafusion"	Output uses standard SQL predicates	Non-INTERSECTS fallback
34	TestTranspileDataFusionDialect	Column-to-column INTERSECTS join	Transpile with default dialect	Output uses standard SQL predicates	Default unchanged
35	TestTranspileDataFusionDialect	Any query	Transpile with dialect="postgres"	Raises ValueError	Invalid dialect

[conradbzura]

Code Review

Blocking

1. is_from_left uses alphabetical heuristic (logical_rule.rs lines 216-238) — Determines which join side a column belongs to by checking if the table alias starts with 'a' or 'l'. Will silently produce wrong results for FROM peaks JOIN genes or any non-trivial aliasing. Must check against join.left / join.right child schemas instead.

2. 17+ eprintln! debug statements in production code — logical_rule.rs (12), optimizer.rs (4), pattern.rs (1). These emit noisy stderr output on every query. Replace with log::debug!/log::trace! or remove.

High

3. Doc comment says enable_logical_rule defaults to false but it defaults to true (lib.rs line 55) — Fix the documentation.

4. ~900 lines of dead code — BinExpandExec, BinnedJoinExec, BinnedSqlExec, pruning.rs, most of cost.rs are never instantiated in any active code path. Remove or feature-gate.

5. Unnecessary unsafe Pin::new_unchecked (sweep_line.rs line 312) — JoinHandle is Unpin, so Pin::new() suffices. The safety comment ("we only poll this once") is also incorrect since this is inside a loop.

Medium

6. choose_bin_size heuristic (logical_rule.rs lines 400-438) — max(end) - max(start) approximates the width of the interval with the largest start position, not the typical width. Could produce poor bin sizes when the widest intervals don't have the largest start coordinates.

7. Logical rule only handles strict </> (logical_rule.rs lines 150-151) — Users writing a.start <= b.end (closed intervals) won't trigger the rewrite. The physical rule's extract_comparison_columns does accept <=/>=, creating an inconsistency.

8. No tests for non-a/l table aliases — Given the is_from_left fragility, this is the most important missing test.

Low

9. Cargo.lock committed — Convention for library crates is to gitignore it.

10. UInt64Array vs UInt32Array inconsistency between bin_expand.rs and sweep_line.rs.