[#2711] fix(spark): Race condition on deferred compressed block initialization

[zuston]

What changes were proposed in this pull request?

Fix race condition on deferred compressed block initialization

Why are the changes needed?

fix #2711

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing unit tests

[github-actions]

Test Results

 3 175 files  ±0   3 175 suites  ±0   6h 53m 44s ⏱️ +44s
 1 238 tests ±0   1 237 ✅ ±0   1 💤 ±0  0 ❌ ±0 
15 689 runs  ±0  15 674 ✅ ±0  15 💤 ±0  0 ❌ ±0 

Results for commit 3d120fb. ± Comparison against base commit f43f66f.

♻️ This comment has been updated with latest results.

[xianjingfeng]

Is this change still needed after using synchronized?

[zuston]

I use the double-lock to ensure the rebuilding happen only once, maybe this is not necessary to make this var sync.

[xianjingfeng]

Could you explain the reason for this issue? @zuston

[zuston]

Could you explain the reason for this issue? @zuston

This bug only happens when the overlapping compression is enabled, this will be modified by the multi threads at the same time

[xianjingfeng]

Could you explain the reason for this issue? @zuston

This bug only happens when the overlapping compression is enabled, this will be modified by the multi threads at the same time

What I don't understand is why block.data would be null in a multi-threaded environment. Even with repeated calls to the rebuildFunction function, block.data is only overwritten, not set to null.

[zuston]

Could you explain the reason for this issue? @zuston

This bug only happens when the overlapping compression is enabled, this will be modified by the multi threads at the same time

What I don't understand is why block.data would be null in a multi-threaded environment. Even with repeated calls to the rebuildFunction function, block.data is only overwritten, not set to null.

Got your thought, I haven't found the root cause.

[zuston]

thanks. @xianjingfeng @jerqi let me merge this