Question about support for validation sets and eval_loss

[jacopocecch]

Hi, I'm currently working on a project where I need to evaluate model performance during training using a validation set (i.e., periodically compute eval_loss), and I'm using Hugging Face's Trainer arguments such as:

--evaluation_strategy steps
--eval_steps 1000
--load_best_model_at_end
--metric_for_best_model eval_loss

However, I've not been able to make it work out of the box.

1. Is validation via eval_dataset currently supported in Tevatron?
2. Is there any recommended or officially supported way to compute eval_loss during training?

Thank you for your time!