emozilla/pg_books-tokenized-bos-eos-chunked-65536

Dataset

7

stars

139

commits

1

linked in READMEs

Oct 7, 2023

updated

README

Dataset Card for "pg_books-tokenized-bos-eos-chunked-65536"

The pg19 dataset tokenized under LLaMA into 64k chunks, bookended with BOS and EOS

Contributors

emozilla

139 commits

emozilla/pg_books-tokenized-bos-eos-chunked-65536

Dataset

7

stars

139

commits

1

linked in READMEs

Oct 7, 2023

updated

README

Dataset Card for "pg_books-tokenized-bos-eos-chunked-65536"

The pg19 dataset tokenized under LLaMA into 64k chunks, bookended with BOS and EOS

Contributors

emozilla

139 commits