Tiny Stories Dataset Reprocessed for Pascal Developers

license: cdla-sharing-1.0

This Dataset contains synthetically generated (by GPT-3.5 and GPT-4) short stories that only use a small vocabulary as described in: https://arxiv.org/abs/2305.07759.

Please reffer to https://arxiv.org/abs/2305.07759 for the original authors and to https://huggingface.co/datasets/roneneldan/TinyStories for the original dataset.

This repository contains a reprocessed version of https://huggingface.co/datasets/roneneldan/TinyStories so Pascal developers can use it. This dataset is composed by 2 CSV that contain:

• Vocabularies
• and tokenized datasets.

Download

This dataset can be downloaded with:

  git clone https://huggingface.co/datasets/schuler/TinyStories4Pascal-Tokenized-v2
  unzip TinyStories4Pascal-v2/tinystories-vocab-3k-cai.csv.zip
  unzip TinyStories4Pascal-v2/tinystories-10k-tokenized3k.csv.zip

How To Use this Dataset with Free Pascal?

To be able to compile the free pascal source code shown below, you'll need Lazarus and CAI Neural API.

The model is created with (2 transformer blocks with 16 heads each):

  var
    CntLayer: integer;
  begin
    Result := THistoricalNets.Create();
    Result.AddLayer([
      TNNetInput.Create(pContextSize, 1, 1),
      TNNetTokenAndPositionalEmbedding.Create(pVocabSize, pEmbedDim),
    ]);
    for CntLayer := 1 to {Layers=}2 do
    begin
      Result.AddTransformerBlockCAI( {Heads=}16, {intermediate dimensions=}4*512, {NoForward=}true, {HasNorm=}true, false);
    end;
    Result.AddLayer([
      TNNetPointwiseConvLinear.Create(pVocabSize, 1),
      TNNetPointwiseSoftMax.Create(1)
    ]);
  end;

You can find a full free pascal source code example at: gpt-3-for-pascal.

How Was The Dataset Transformed From Its Original Shape?

The original dataset was transformed using with the source code shown below.

A text file was created with:

wikids = load_dataset("roneneldan/TinyStories", split="train")

MIN_TRAINING_SEQ_LEN = 20
prepared_ds = []
row_cnt = 0
for ds_row in wikids:
  row_cnt = row_cnt + 1
  new_row = ds_row['text'].strip(" '\"")
  new_row = new_row.replace(' .', '.').replace(' ,', ',').replace(' !', '!').replace(' ?', '?').replace(' ;', ';').replace(' :', ':').replace(" '", "'")
  new_row = new_row.replace('<unk>', '').replace('  ', ' ')
  # remove non ascii characters from new_row
  new_row = ''.join([i if (ord(i) < 128) and (ord(i) > 31) else '' for i in new_row])
  # remove any linefeed
  new_row = new_row.replace('\n', '')
  new_row_len = len(new_row)
  if ( new_row_len > MIN_TRAINING_SEQ_LEN ):
    prepared_ds.append(new_row)
  if row_cnt % 100000 == 0:
    print(len(prepared_ds), "loaded rows.")
print("We have", len(prepared_ds), "strings in the dataset out of a total of", row_cnt,'.')

def save_dataset(dataset, filename):
    with open(filename, 'w') as f:
      for item in dataset:
        f.write("%s\n" % item)

save_dataset(prepared_ds,'tinystories.txt')

The vocabulary was created with:

var
  X: TNeuralTokenizer;
begin
  X := TNeuralTokenizer.Create;
  X.FitOnFile('datasets/tinystories-10k.txt', 3000);
  X.SaveToFile('datasets/tinystories-vocab-3k-cai.txt');
  X.Free;
end;

The tinystories-100k-tokenized3k.csv file was created with:

var
  X: TNeuralTokenizer;
begin
  X := TNeuralTokenizer.Create;
  X.LoadVocabularyFromFile('datasets/tinystories-vocab-3k-cai.txt');
  X.TokenizeFileToCsv('datasets/tinystories-100k.txt','datasets/tinystories-100k-tokenized3k.csv');
  X.Free;
end;