Andrew Lamb Weekly-ish Open Source plan - 2026-01-05

[alamb]

This is my weekly plan, mostly for my own organizational need. I am making it public in the hopes that helps others to see what I am working on -- also I spend so much time in github the interface is very familiar to me and I can cross link all the issues I am working

(it is also my excuse as to why I haven't reviewed many good looking PRs)

Notes to myself: a duplicate entry unchecked means I need to go back and re-review

PR review queue (rough order)

• Remove parquet arrow_cast dependency arrow-rs#9077
• Row group limit pruning for row groups that entirely match predicates #18868
• Allow struct field access projections to be pushed down into scans #19538
• Refactor PartitionedFile: add ordering field and new_from_meta constructor #19596
• add specialized InList implementations for common scalar types #18832
• Incremental group emission in HashAggregate #19562
• feat: Prune complex/nested predicates via statistics propagation #19609
• feat: TPC-DS support clflushopt/tpchgen-rs#212
• [EPIC] Adaptive Query Execution (AQE) datafusion-ballista#1359 (comment)
• Fix IPC roundtripping dicts nested in ListViews arrow-rs#9126
• Add BooleanBufferBuilder::extend_trusted_len arrow-rs#9137
• perf: improve field indexing in JSON StructArrayDecoder (1.7x speed up) arrow-rs#9086
• perf: improve calculating length performance for view byte array in row conversion arrow-rs#9080
• Add BatchAdapter to simplify using PhysicalExprAdapter / Projector to map RecordBatch between schemas #19716
• perf: optimize left function by eliminating double chars() iteration #19571
• help CMU at feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations  #18806
• Cache PlanProperties, add fast-path for with_new_children #19792
• Avoid panic on Date32 overflow arrow-rs#9144
• Refactor cache APIs to support ordering information #19597
• Allow dropping qualified columns #19549
• [parquet] Add row group index virtual column arrow-rs#9117
• Record sort order when writing Parquet with WITH ORDER #19595
• [Parquet] perf: preallocate capacity for ArrayReaderBuilder arrow-rs#9093
• [DRAFT] Extension Type Registry Draft #18552
• add nested nullable field support for CachedArrayReader arrow-rs#9066
• Allow extensions to arrow-json decoder and include an extension for variant arrow-rs#9021
• docs: auto generate metrics documentation #19367
• Extend comparison support for Union arrays against an opaque array arrow-rs#8881
• Timezone aware extract SQL expression #18990
• Support "pre-image" for pruning predicate evaluation #18789
• Support reverse parquet scan and fast parquet order inversion at row group level #18817
• fix: TableScan should recurse into provider logical plan in map_children #19282
• feat(memory-tracking): expose API to NullBuffer, ArrayData, and Array arrow-rs#8918 (comment)
• [DRAFT] Extension Type Registry Draft #18552
• feat: implement GroupArrayAggAccumulator attempt 3 #17915
• Row group limit pruning for row groups that entirely match predicates #18868 (rereview)

Projects I am supporting actively (high on my priority list)

• Improve DataFusion ClickBench performance (via pushdown predicates): Enable parquet filter pushdown (filter_pushdown) by default #3463
• Release arrow-rs 57.2.0: Release arrow-rs / parquet Minor version 57.2.0 (December 2025) arrow-rs#8465
• DataFusion object store caching improvements with @BlakeOrth [EPIC] ListingTable object store usage improvements #17214

Projects on my backlog

These are ones I would like to support but don't have the capacity at the moment to push, in relative order

• Help integrate Variant with @friendlymatthew [EPIC] Support VARIANT type for unstructured data #16116
• Epic: Join Order Enumeration #18249 from @NGA-TRAN
• feat: support invoking table functions with tables rather than a single expression #18535

PRs that look great but need a thorough review (looking for help here 🎣 from anyone else)

• external tables for multiple locations: feat(cli): support external tables on multiple locations #17702
• writing REE arrays to parquet: Support writing RunEndEncoded as Parquet arrow-rs#8069

[alamb]

Next chunk:

• Andrew Lamb Weekly-ish Open Source plan - 2026-01-19 #19886