Andrew Lamb Weekly-ish Open Source plan - 2025-11-17

[alamb]

This is my weekly plan, mostly for my own organizational needs (as I am dropping too many things). I am making it public in the hopes that helps others to see what I am working on -- also I spend so much time in github the interface is very familiar to me and I can cross link all the issues I am working

(it is also my excuse as to why I haven't reviewed many good looking PRs)

I will update this list as I run into more things

Notes to myself: a duplicate entry unchecked means I need to go back and re-review

PR review queue (rough order)

• Speed up collect_bool and remove unsafe, optimize take_bits, take_native for null values arrow-rs#8849
• Add ability to skip or transform page encoding statistics in Parquet metadata arrow-rs#8797
• Removed incorrect union check in enforce_sorting and updated tests #18661
• Refactor InListExpr to support structs by re-using existing hashing infrastructure  #18449
• [Variant] Enforce shredded-type validation in shred_variant arrow-rs#8796 (review)
• Adds prefix filtering for table URLs #18780
• Allow Users to Provide Custom ArrayFormatters when Pretty-Printing Record Batches arrow-rs#8829
• Add comparison support for Union arrays arrow-rs#8838
• Add merge and merge_n kernels arrow-rs#8753
• Add support for Union types in RowConverter arrow-rs#8839
• Move GuaranteeRewriter to datafusion_expr #18821
• Fix Partial AggregateExec correctness issue dropping rows #18712
• Adds memory-bound DefaultListFilesCache #18855
• FFI: return underlying trait type when converting from FFI structs #18672
• add specialized InList implementations for common scalar types #18832
• Optimize planning for projected nested union #18713
• Support reverse parquet scan and fast parquet order inversion at row group level #18817
• [Variant] Improve variant_get performance on a perfect shredding arrow-rs#8887
• Add union to opaque comparisons arrow-rs#8896
• Add validated constructors for UnionFields arrow-rs#8891
• chore: update Repartition DisplayAs to indicate maintained sort order #18673
• Enable Parallel Aggregation for Non-Overlapping Partitioned Data #18826
• optimizer: Support dynamic filter in MIN/MAX aggregates #18644
• [DRAFT] Extension Type Registry Draft #18552
• move projection handling into FileSource #18627
• Make Parquet SBBF serialize/deserialize helpers public for external reuse arrow-rs#8762
• feat: implement GroupArrayAggAccumulator attempt 3 #17915
• feat: Support recursive queries with a distinct 'UNION' #18254
• Test datafusion with new sqlparser: Eliminating whitespace from the parser logic datafusion-sqlparser-rs#2076 (comment)
• Refactor state management in HashJoinExec and use CASE expressions for more precise filters #18451
• Refactor state management in HashJoinExec and use CASE expressions for more precise filters #18451
• Add relation planner extension support to customize SQL planning #17843

What am I personally working on now

• Internal error: Physical input schema should be the same as the one converted from logical input schema. #18337 (and related fallout)

Projects I am supporting actively (high on my priority list)

• Next DF release: Release DataFusion 51.0.0 (Nov 2025) #17558 with @xudong963
• Next arrow release; Release arrow-rs / parquet Minor version 57.1.0 (November 2025) arrow-rs#8464
• Improve DataFusion ClickBench performance: [EPIC] Make DataFusion the top of the ClickBench Parquet leaderboard #18489
• Adaptive parquet predicate pushdown evaluation faster with @hhhizzz [Parquet] Adaptive Parquet Predicate Pushdown arrow-rs#8733
• Low level arrow bit manipulation fast with @rluvaton feat: add apply_unary_op and apply_binary_op bitwise operations arrow-rs#8619
• DataFusion object store requests go faster with @BlakeOrth [EPIC] ListingTable object store usage improvements #17214
• Release object store 0.13.0: Release object store 0.13.0 (breaking) - Target Nov 2025 arrow-rs-object-store#367

Projects on my backlog

These are ones I would like to support but don't have the capacity at the moment to push, in relative order

• Help integrate Variant with @friendlymatthew [EPIC] Support VARIANT type for unstructured data #16116
• Epic: Join Order Enumeration #18249 from @NGA-TRAN

PRs that look great but need a thorough review (looking for help here 🎣 from anyone else)

• Examples of extending SQL syntax #17824 from the @theirix
• external tables for multiple locations: feat(cli): support external tables on multiple locations #17702
• relation extension planner: Add relation planner extension support to customize SQL planning #17843
• writing REE arrays to parquet: Support writing RunEndEncoded as Parquet arrow-rs#8069

[alamb]

• Next Andrew Lamb Weekly-ish Open Source plan - 2025-11-24 #18888