Skip to content

wrong results for null count of nullif kernel #9085

@alamb

Description

@alamb

Describe the bug
When calling nullif(a, b) and b has no nulls, sometimes the null count of the returned buffer is incorrect

To Reproduce
Add this to the nullif_fuzz test:

@@ -502,15 +529,15 @@ mod tests {
         let mut rng = rng();

         let arrays = [
-            Int32Array::from(vec![0; 128]),
-            (0..128)
-                .map(|_| rng.random_bool(0.5).then_some(0))
+            Int32Array::from(vec![0; 1024]), // no nulls
+            (0..1024) // 50% nulls
+                .map(|_| rng.random_bool(0.5).then_some(1))
                 .collect(),
         ];

         for a in arrays {
-            let a_slices = [(0, 128), (64, 64), (0, 64), (32, 32), (0, 0), (32, 0)];
-
+            //let a_slices = [(0, 128), (64, 64), (0, 64), (32, 32), (0, 0), (32, 0)];
+            let a_slices = [(0, 128), (5, 800), (33, 53), (64, 64), (0, 64), (32, 32), (0, 0), (32, 0)];
             for (a_offset, a_length) in a_slices {
                 let a = a.slice(a_offset, a_length);

@@ -518,12 +545,17 @@ mod tests {
                     let b_start_offset = rng.random_range(0..i);
                     let b_end_offset = rng.random_range(0..i);

+                    // b with 50% nulls
                     let b: BooleanArray = (0..a_length + b_start_offset + b_end_offset)
                         .map(|_| rng.random_bool(0.5).then(|| rng.random_bool(0.5)))
                         .collect();
-                    let b = b.slice(b_start_offset, a_length);
+                    let b_sliced = b.slice(b_start_offset, a_length);
+                    test_nullif(&a, &b_sliced);

-                    test_nullif(&a, &b);
+                    // b with no nulls
+                    let b = make_array(b.into_data().into_builder().nulls(None).build().unwrap());
+                    let b_sliced = b.as_boolean().slice(b_start_offset, a_length);
+                    test_nullif(&a, &b_sliced);
                 }
             }
         }

Expected behavior
The test should pass

Additional context
I found this while debugging issues in applying the same pattern in #8996 to this kernel.

I am pretty sure I introduced this in #8996

Basically, there is an implicit assumption in the nullif kernel that the op will only be called on words with bits that are completely contained within the offset/length of the input arrays, which is not true when creating using word aligned Vecs

Metadata

Metadata

Assignees

No one assigned

    Labels

    arrowChanges to the arrow cratebugparquetChanges to the parquet crate

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions