Mean score~: -5.37444

{
  "text": "Once upon a time...",
  "pos": -5.37444
}

roughly 3.98 billion tokens depending on your tokenizer

asked mistral something along the lines of "how good is this text for pretraining" and the output is a dataset that token for token is better quality (produces higher benchmarking model) than minipile, even though the minipile effort was incredible and i think they're great