Add Union type casting support to `arrow-cast`

[friendlymatthew]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Arrow's cast kernel currently doesn't support casting Union types to other types. This blocks useful operations like comparing Union columns with scalar values in query engines like Datafusion (see apache/datafusion#18825)

I propose the following behavior:

When casting Union(Int32, Utf8) to Int32:

• extract values from the int32 variant
• non-matching variants (Utf8) become Null

There's also an intermediary cast that could occur, for instance when casting Union(Int32, Utf8) to Int64:

• extract values from the int32 variant
• cast them if needed (e.g., Int32 -> int64)
• non-matching variants (Utf8) become null

This implementation is quite greedy: it finds the matching variant by exact type match first, then by cast-compatibility. (Users can determine precedence by a Union variant's type_id).

Motivation:

• Related Coerce union pydantic/datafusion#49
• Related Extend comparison support for Union arrays against an opaque array #8881

[friendlymatthew]

Curious what you think @alamb, this might be too crazy but this seems to work really well in our pydantic fork

[alamb]

I think this sounds like a specialized cast kernel -- the arrow-rs cast kernel is pretty dumb and just does what it is told. I think the logic you describe seems reasonable but I can imagine other semantics as well

Maybe a new kernel like union_cast 🤔 (that DataFusion could potentially call) ?

[Rafferty97]

I think this could be decomposed into simpler pieces. For the case of just wanting to extract a particular variant from an enum without any further transforms/casts, I think this could be relatively simple:

• For a sparse union, I think it's just a matter of extracting the right child array and adjusting the validity bitmap
• For a dense union, it would be similar but you'd need to scatter the actual values

@alamb If you think this could be a useful lego piece I'd be interested in trying my hand at an implementation

[alamb]

It looks like @friendlymatthew made a PR yesterday:

• Add support to cast from UnionArray #9544

@alamb If you think this could be a useful lego piece I'd be interested in trying my hand at an implementation

yes I do

For "scatter" I think you can use the pre-existing interleave kernel