training from scratch

[shouryaverma]

Hello thank you for a great paper!

I can see the inference and evaluate files and notice you have provided the checkpoint for the best model. But there are no steps to train a model.

How can I train a new model from scratch? Is there a missing trainer.py file. Or maybe I overlooked something.

Thank you!

[VicFisher]

Hello Shourya,

Thank you for your interest in our work and for your kind words about the paper!

We appreciate your question about training a new model from scratch. While we included a training demo in a previous version, we omitted the full training script due to the complexity of the pretraining process and the numerous parameters involved. However, we’re happy to assist! We can provide the training script (train.py) and the cached dataset to help you get started. Please let us know if you’d like us to upload these, and we’ll make them available as soon as possible.

In the meantime, could you clarify whether you’re looking to replicate the pretraining process or fine-tune the model? This will help us tailor our guidance to your needs.

Thank you for your patience, and feel free to reach out with any further questions!

Best regards,
Yu JunLin.

[shouryaverma]

Hello Yu,

Thanks for getting back so quickly.

Yes I wanted to pretrain the model from scratch and run scaling experiments to test how much training data would be needed to achieve high accuracy. If there were numerous parameters involved then I can run a grid search to find the optimal parameters.

Please let me know if its feasible to provide the pretraining configuration files and data. I would really appreciate the help!

I hope the experiments I suggested are possible to run with the updated training script and cached dataset you may provide.

Thank you!

Shourya