Difference between RedPajama-Data-1T, RedPajama-Data-V2, RedPajama-Data-V1

[konradipipan]

Is 1T version basically V1? If so, is the HF version of V1 (1T) already deduplicated are ready to be used?

[mauriceweber]

Hi @konradipipan -- You are correct, RedPajama-Data-1T on HF corresponds to v1. This dataset is not deduplicated. If you want a deduplicated version, you can check out SlimPajama, which is a version of RPv1 which is cleaned and deduplicated across dataset slices with MinHashLSH.