[BUG] Investigate non-determinism in evo2 recipe data preprocessing script and/or test #1493
Description
BioNeMo Framework Version
main
Bug Description
The recent PR #1492 addressed this issue by relaxing the constraints. However the intermittent CI failure that lead to the number of splits not being the same in every run highlights a bug where randomness is leaking into the process somewhere.
Steps to Reproduce
See #1492, reset the test to the previous strict equality and there are intermittent (rare) failures.
Error Messages and Logs
Docker Image
No response
System Information
Environment Details:
- OS: [e.g., Ubuntu 20.04]
- CPU: [e.g., Intel i9-12900K]
- RAM: [e.g., 64GB]
GPU Details:
- GPU Model: [e.g., NVIDIA RTX 4090]
- GPU Memory: [e.g., 24GB]
- CUDA Version: [e.g., 12.1]
- CUDA Driver: [e.g., 525.85.05]
- cuDNN Version: [e.g., 8.9.0]
Additional Context
No response