Hello Ruard,
In the file evaluation/openmolcas/evaluate_model_loss.py I think there is an issue on how the loss is being calculated.
On line 21 and 63 I think one should be dividing by the len(indices) to account for different training/validation/test set sizes.
All the best,
João