reddit data sources #7
Description
Hey @taisazero I saw you were working on the reddit part of this script. I'm interested in including more natural dialogue data in future open source models, so I thought I would share some links to reddit data that might help you.
- this dataset has a list of nice subreddits https://huggingface.co/datasets/stanfordnlp/SHP
- there are other sources of reddit data, not just the api pushshift.io. Some of them might be faster to process!
- Up to 2023 (2TB) on academictorrents
- Up to 2020 on bigquery
Thanks for your work here.