Question about data generation process for olmo-mix-1124.

[WwwwYz666]

❓ The question

Hi allenai team!

I was exploring the allenai/olmo-mix-1124 dataset and noticed references to preprocessing scripts in the repository. Specifically, I found prepare_memmap_dataset.py which seems related to data preparation.

Could you kindly clarify:

How was the olmo-mix-1124 dataset generated?
Is prepare_memmap_dataset.py the primary script used? If so, could you provide a basic reference command for generating a similar dataset?
If other scripts/configurations were involved, would you mind sharing any pointers?
Thank you for your work and guidance!

[RobertXWL]

Hi, I’m also very interested in this topic and would like to follow up.

It would be extremely helpful if you could provide:

• The processing scripts or code snippets used to generate the dataset.
• Any configuration files, options, or parameters that were important in the preprocessing stage.
• A brief description of the end-to-end data preparation workflow.

This information would make it easier for the community to reproduce or build upon your work.

Thank you in advance for considering this request!