Dataset can be found here: https://huggingface.co/datasets/enryu43/twitter100m_tweets/viewer/default/train

To download the hf dataset see below, I made a python file to download it since I had issues with other approaches. Apparently I deleted it though since I can't find it anymore https://huggingface.co/docs/hub/datasets-downloading

By default the hugging face files are downloaded to ~/.cache/huggingface/hub

Running it

• You will need lid.176.bin in src/
• dataset-0.parquet contains the language classified data