Make training deterministic

[aditya0by0]

Using seed_everything(seed) helps with reproducibility by setting seeds for Python's random, NumPy, and PyTorch (both CPU and CUDA), and it also sets:

1. random.seed(seed) — Python’s built-in random module.
2. np.random.seed(seed) — NumPy's random number generator.
3. torch.manual_seed(seed) — PyTorch’s RNG on CPU.
4. torch.cuda.manual_seed(seed) — PyTorch’s RNG on current GPU.
5. torch.cuda.manual_seed_all(seed) — For all GPUs.
6. torch.backends.cudnn.deterministic = True — Forces cuDNN to use deterministic algorithms.
7. torch.backends.cudnn.benchmark = False — Disables the autotuner that selects the fastest convolution algorithm (introduces randomness).

However, this doesn't imply that Trainer(deterministic=True) is set. The deterministic flag in the Trainer does more — it enforces deterministic algorithms during training and can catch known non-deterministic ops (especially on GPU), which is useful for debugging reproducibility issues.

✅ Recommendation: Use both together for maximum reproducibility:

seed_everything(42)
trainer = Trainer(deterministic=True, ...)

Pytorch-lightning sets seed_everything to 0 by default, if no seed is provided but not the trainer's deterministic flag which is set to None by default.

Please check the below links:

• https://lightning.ai/docs/pytorch/stable/common/trainer.html#reproducibility
• training on gpu becomes non-deterministic Lightning-AI/pytorch-lightning#11250

[aditya0by0]

This PR will complete this issue #12 in its entirety.

[aditya0by0]

@sfluegel05, I tested this by running the same training configuration twice using the same seed.

• Before setting deterministic=True:
  The results showed slight variations:

  • Run 1: https://wandb.ai/chebai/chebai/runs/5p6gd1wz
  • Run 2: https://wandb.ai/chebai/chebai/runs/ztfi8841

• After setting deterministic=True in the Trainer:
  🟡 The results were identical, with all metrics matching exactly down to the last decimal:

  • Run 1: https://wandb.ai/chebai/chebai/runs/bq12dkhn
  • Run 2: https://wandb.ai/chebai/chebai/runs/xwouq71m

✅ This confirms that setting deterministic=True in addition to seed_everything() is essential for full reproducibility in PyTorch Lightning.

[sfluegel05]

That's really good to know, thanks for finding this out.