Checkpoints for best 1B models at 8K context

[Thiggel]

Dear authors,

congrats for this amazing research!

I would love to experiment with nGPT, but I only have access to limited compute budget. Therefore, I am asking whether you could provide checkpoints for your baseline and nGPT models at 1B params at 8K context.

This would immensely help me in my own research.

Thanks a lot in advance.

Best regards,
Filipe Laitenberger

[Thiggel]

Dear authors,

In the paper, you mention that you use the Llama-2 tokenizer. Did you also train a modified version of Llama-1b, so that the resulting checkpoint could be loaded into a normalized Llama-2b model?

The other checkpoints have not been uploaded yet, or did I miss them (I saw you reacted to my last question but did not know whether this meant you've already uploaded them).

Again, thanks a lot for your help as the checkpoints would free me from re-training a model from scratch (which I simply cannot do at my university).

Best regards

[iloshchilov]

This code was not used to train models in the paper and we do not have any checkpoint for 1 B or for 8K which are compatible with this code.

[Thiggel]

Dear @iloshchilov,

thanks for your answer. Do you have any other 1B with 8K context checkpoints? Even if they are not compatible with this code, they would immensely help me, as I could write the code myself.

Best regards

[iloshchilov]

We do have checkpoints for 1B and 8K which were used for the paper but to publish them we would need an approval from the legal people of the company (because that corresponds to publishing a model which uses a dataset, we have not requested such an approval) and I am not sure how the model can be used without the corresponding code which was used for its training (which uses an internal library which is not open sourced).

[Thiggel]

Thanks for this clarification. Would it nonetheless be possible to ask for the approval and publish the internal library based checkpoints? As it was trained on a public dataset there shouldn't be a problem with this right?

Based on the checkpoint, it should be easily possible to reconstruct the model, even with e.g. PyTorch.

I was trying to train nGPT using a 1B param model with tiny context (128) but even for that my resources simply do not suffice. I would be immensely grateful for your checkpoint, because it would be incremental to my research, otherwise I would not be so persistent about it.