Add `ListViewArray` and `LargeListViewArray`

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Recently two new types were added to the Arrow format that make it more suitable for certain types of operations on Lists

Specifically when doing filtering / take with List data, creating a new ListArray or LargeListArray requires copying the underlying lists to a new, packed buffer. The "ListView" was designed to solve this limitation and recently added to the Arrow spec.

Describe the solution you'd like
I would like to implement ListViewArray and LargeListViewArray following the spec:
The spec: https://arrow.apache.org/docs/format/Columnar.html#listview-layout

Initially, I would suggest we get the basic types in place:

• DataType
• Array implementations and layout and basic construction

Then as follow on PRs, add support to key kernels:

• cast (to/from ListAray / LargeListArray)
• filter: Support more operations on ListView #8645
• take: Support more operations on ListView #8645

Describe alternatives you've considered

Additional context
This is similar in spirit to the StringViewArray and BinaryViewArray described in #5374

Tasks:

• Add DataType::ListView and DataType::LargeListView #5492
• Add ListViewArray and LargeListViewArray implementation and layout and basic construction #5501
• Roundtripping dicts nested in ListView not working #9125
• Improve casting for listview types #9264
• Support ListView in arrow-json #9340
• Support ListView in sort kernel #9341
• Add native ListView support for interleave kernel #9342
• Support ListView in length kernel #9343
• Support roundtrip ListView in parquet arrow writer #9344

[Kikkon]

Hi @alamb

I'm interested in the progress of this issue regarding the addition of ListViewArray and LargeListViewArray to arrow-rs. I've been following the discussion and would like to know if there have been any updates or if there's anything specific I can assist with to help move this forward.

[alamb]

Hi @alamb

I'm interested in the progress of this issue regarding the addition of ListViewArray and LargeListViewArray to arrow-rs. I've been following the discussion and would like to know if there have been any updates or if there's anything specific I can assist with to help move this forward.

Hi @Kikkon -- that is great news. 🙏

There have been discussions related to implementing StringView and BinaryView as part of #5374 and I expect that work to begin shortly (within a month maybe?).

I don't know of any similar work afoot for ListViewArray but I suspect we could follow a very similar pattern to #5374 (e.g. implement the basic array structure first, followed by cast, take, and filter kernels)

Does that make sense?

[Kikkon]

Hi @alamb I have already created some preliminary issues: #5492 and pull requests #5493 . Regarding the subsequent tasks, if time permits, I can also try it.

[alamb]

Thank you @Kikkon -- that sounds awesome

[Kikkon]

@alamb Since #5492 has already been merged, I've created an issue #5501 to keep track of the progress. Is there anything that needs to be added?"

[alamb]

@alamb Since #5492 has already been merged, I've created an issue #5501 to keep track of the progress. Is there anything that needs to be added?"

Thanks @Kikkon

I added #5501 to the top of this ticket. In terms of next steps, I think the list of items we are finding on #5374 is worth looking at (e.g. IPC, support for filter/take, etc). We'll keep that list updated

[adriangb]

Naive question about ListViewArray: does it do interning like StringViewArray does? That is, if I have a column with data [ [1, 2], [2, 3], [2, 3] ] what would the actual data layout be?

Edit: the answer is yes, based on the arrow docs linked here. Second example shows it.