Reconsider equality of `ScalarValue`s for encoded types (REE/Dictionary)

[Jefffrey]

Is your feature request related to a problem or challenge?

@brancz brought up a point on how we might want to not consider the index/key types of REE/Dict scalarvalues when checking equality:

• Add ScalarValue::RunEndEncoded variant #19895 (comment)

Code reference:

datafusion/datafusion/common/src/scalar/mod.rs

Lines 727 to 739 in 8c478e9

	(Dictionary(k1, v1), Dictionary(k2, v2)) => {
	// Don't compare if the key types don't match (it is effectively a different datatype)
	if k1 == k2 { v1.partial_cmp(v2) } else { None }
	}
	(Dictionary(_, _), _) => None,
	(RunEndEncoded(rf1, vf1, v1), RunEndEncoded(rf2, vf2, v2)) => {
	// Don't compare if the run ends fields don't match (it is effectively a different datatype)
	if rf1 == rf2 && vf1 == vf2 {
	v1.partial_cmp(v2)
	} else {
	None
	}
	}

Describe the solution you'd like

Consider ignoring the index/key types for scalarvalue equality of REE/dictionary values. What knock-on effects will this have? What does it enable for us?

Describe alternatives you've considered

Don't do this if we prefer to stick to strict equality.

Additional context

No response

[kumarUjjawal]

Should equality treat key/run-end integer width as non-semantic (so logically identical values compare equal), or keep current strict type/field must match behavior? If we chose the former, do we want that change in global PartialEq/Hash, or via a separate logical equality helper?

[Jefffrey]

The purpose of this issue is to explore which should be done. Currently we do strict equality, but if we want to change this we'd need to see if there are benefits (are there things not happening in the code because of this strict equality) and if there are other knock-on effects.

[kumarUjjawal]

I added a couple of small tests to gather evidence:
• Dictionary DISTINCT: DistinctCountAccumulator currently treats dictionary scalars as distinct if they
differ only by dictionary key width (e.g. Dictionary(Int8, "a") vs Dictionary(Int32, "a")), so COUNT(DISTINCT ...)
would count them separately if mixed widths ever reach the accumulator.
• REE DISTINCT: same for RunEndEncoded differing only by run-end integer width (e.g. Int16 vs Int32) still
counts as distinct today.
• Execution realism (Union): I tried to force mixed dictionary key widths through a physical UnionExec. Current
behavior rejects/doesn’t align schemas (it panics internally during alignment), so at least via UNION it’s hard for
mixed key widths to arise under “normal” plan construction.

Net: strictness is observable and can affect DISTINCT semantics if mixed widths appear, but some common execution
paths (like UNION) prevent mixed-width dictionary streams today.

[Jefffrey]

I added a couple of small tests to gather evidence: • Dictionary DISTINCT: DistinctCountAccumulator currently treats dictionary scalars as distinct if they differ only by dictionary key width (e.g. Dictionary(Int8, "a") vs Dictionary(Int32, "a")), so COUNT(DISTINCT ...) would count them separately if mixed widths ever reach the accumulator. • REE DISTINCT: same for RunEndEncoded differing only by run-end integer width (e.g. Int16 vs Int32) still counts as distinct today. • Execution realism (Union): I tried to force mixed dictionary key widths through a physical UnionExec. Current behavior rejects/doesn’t align schemas (it panics internally during alignment), so at least via UNION it’s hard for mixed key widths to arise under “normal” plan construction.

Net: strictness is observable and can affect DISTINCT semantics if mixed widths appear, but some common execution paths (like UNION) prevent mixed-width dictionary streams today.

These usecases sound like they should have been handled by type coercion indeed

[kumarUjjawal]

Any feedback on how should I proceed next, anything to check?

[Jefffrey]

Unless we find a motivating usecase I don't think we would proceed with this issue