[!IMPORTANT]

• ✨ Accepted at IJCNLP-AACL Findings 2025 ✨
• This repository provides resources and code and key results for our Regional-TinyStories framework, which extends the TinyStories approach (Eldan & Li, 2023) to Hindi, Marathi, and Bangla
• Our framework enables training and inference of Small Language Models (SLMs) ranging from 5M to 150M parameters, which are subsequently employed as proxies for a variety of comparative analyses

[!NOTE]

A special thanks to

• TensorDock for providing compute! Check them out for easy-to-deploy and affordable GPU/CPU VMs 💚
• Microsoft for inspiring us with their original TinyStories paper 💙
• Sarvam, SUTRA, and Andrej Karpathy for their open-source efforts ❤️

Clone

git clone https://github.com/nirvan840/Vizuara-TinyStories-Regional.git

Requirements

pip install g4f[all] aiolimter transformers datasets huggingface_hub sentencepiece tiktoken wandb tqdm torch numpy 

📚 Table of Contents

• 🗂️ Dataset Generation

  • ✍️ Preparing Prompts

  • 💬 Prompting an LLM

• ⚙️ Training SLMs

  • 🔤 Tokenizing Data

  • 🏋️ Training the Model

• 🔍 Inference and Evaluation

  • 🤖 SLM Inference

  • 📊 Evaluate Inference/Stories

• 📈 Results

  • ⚙️ Hyperparameter Comparisons

  • 💡 Inference Examples

  • 🆚 Synthetic VS Translated

  • 🆚 Tokenizer Comparison

  • 🔎 Language Complexity

  • ✅ A Fitting Use Case

• 💰 Costs

  • ⏱️ Training Time and Costs

🗂️ Dataset Generation

[!WARNING] This repository provides code to generate data by making API calls to SOTA models (4o, 4o-mini, Gemini-flash-1.5, etc.) using the GPT-4-free (G4F) repository. This repository is provided for research purposes only. We do not intend to promote using this repository for large-scale dataset generation; respect all terms of service for any API or model you use. Ensure appropriate attribution and licensing for any generated content

[!NOTE]

• Our datasets for Hindi, Marathi and Bangla, generated using GPT-4o-mini, are open-sourced on our HF
• Translated versions (Hindi and Bangla) of Microsoft's TinyStories dataset can also be found on our HF
• Translated versions (our Hindi ➡️ Bangla ; Hindi ➡️ Marathi ; Bangla ➡️ Hindi) will be soon on our HF

✍️ Preparing Prompts

• Each prompt is generated by sampling a unique set of a noun, a verb, an adjective and a feature
• To modify the list of nouns, verbs, etc., please modify .txt files at prompting/prompt_gen/<lanauge>
• Prompt templates/complexities can be referred to/modified through prompting/prompt_gen/create_prompts.py.
• We compare various complexities in our paper and find 2+ to be optimal
• Unique (sampling is unique and not random) prompts can be generated by running generate_prompts.py
• Generated prompts are written to a .json file. Sharding the file is recommended (below)

To generate prompts please run:

python prompting/prompt_gen/generate_prompts.py

💬 Prompting an LLM

• Prompts are read from the .json file/shards and "sent" to the specified LLM (using G4F)
• Multithreaded API calls result in a max speed of 100 stories/min for GPT-4o-mini and GPT-4o (occasionally).
• It is recommended number of threads = number of 4 x vCPUs
• Each thread writes to a common file. Once every 10% of total progress
• Optimal config: 16 vCPUs, running 4 sessions concurrently (one for each shard), each with 16 threads
• Please look into prompting/make_requests.py for customizing the prompting schema
• Generated stories are written to .json files
• It is recommended to upload these files to HF for seamless integration while training models (below)
• Please look into prompting/request_helper.py for a detailed look into the process
• API/LLM is prompted until a valid story is generated for each prompt
• Various regex (data cleanup) features

Optimal prompt complexity/template:

f```Write a short story in {language} (in Devanagari script) suitable for 5-to-7-year-old children.
Use simple, easy-to-understand words and limit the story to 3-4 short paragraphs (around 200-300 words).
The story should feature a clear beginning, middle, and end. Incorporate the verb "{verb}", the noun "{noun}", and the adjective "{adjective}" naturally into the story.
The story should also integrate the conclusion/tone "{feature1}" through actions and outcomes without directly stating the tone (e.g., do not use "खुश" or similar words explicitly).
Remember to only use simple words and keep the story short!

Return the output as a JSON dictionary in the following format:
{
    "story": "your_generated_story"
}```

To start the data generation process, please run the script:

python prompting/make_requests.py

TIP💡: To run data generation in the background (detached VM session):

tmux new -s session_name

⚙️ Training Small Language Models (SLMs)

[!IMPORTANT]

• It is essential that data is tokenized correctly story (read below)
• Lower end GPUs T4 (Collab), P100 (Kaggle) can be used to train models in <24hrs on our datasets!

[!NOTE]

• We utilize Andrej Karpathy's nanoGPT repository (with modifications) to train models
• Our training script supports multi-GPU training (DDP), progress (TQDM) and logging support (WANDB), along with easy customizability

🔤 Tokenizing Data

• The .json files uploaded to HF in the previous stage server as our dataset
• The entire dataset is tokenized before training. Token IDs are stored in .bin files
  • The dataset to be tokenized can be chosen as per training-inference/data/prepare.py line 38-46
  • To tokenize a custom HF dataset, please look into lines 59-61 & 112-133
  • The .bin files must be appropriately placed in a folder in training-inference/data/
  • This folder must be specified in config.py under the dataset variable
  • Use training-inference/data/decode_data.py to decode and print first 500 tokens from a .bin file
  • Ensure that the decoded tokens follow the format: <bos token> story1 <eos token> <bos token> story2 <eos token>...
• Tokenization is carried out by the script training-inference/data/prepare.py
• We provide direct support for the following tokenizers:
  • Sarvam (sarvam-1) (Indic)
  • Sutra (mlt256-v2) (Indic)
  • Tiktoken (GPT-2) (English)
• We provide easy support for any additional tokenizers available on HF.
  • Specify the new tokenizer along similar lines astraining-inference/data/prepare.py line 19-35
  • For this to work, HF tokenizers must have valid eos and bos tokens

To tokenize an HF dataset, run the script:

python training-inference/data/prepare.py

🏋️ Training the Model

• Training can be resumed for locally saved models as well as those from Vizuara-HF!
• Changes to the training configuration can be easily made through training-inference/config.py
• Model weights are checkpointed every eval_iters and saved at training-inference/out/ as .pt files

To start training, run:

# Single GPU
python training-inference/train.py training-inference/config.py

# Multi GPU (DDP)
torchrun --standalone --nproc_per_node=num_gpus training-inference/train.py training-inference/config.py

Automated multi-config training:

chmod +x training-inference/utils/automate-training.sh
./training-inference/utils/automate-training.sh

🔍 Inference and Evaluation

[!IMPORTANT]

• All evaluations are performed by SOTA LLMs, which may be "biased" towards a language
• Thus, an 8/10 Hindi story might not be equivalent in "quality" to an 8/10 Bangla story
• It is crucial to understand that as these models evolve, "ratings"/results will drift as compared to today

[!NOTE]

• Given the small size of the models, CPU inference is supported!
• It is crucial to ensure tokenization occurs correctly (refer below)!
• Data generation scripts are repurposed for evaluation of stories produced by SLMs

🤖 SLM Inference (Local or HF)

• Refer to the sample settings at the bottom of training-inference/config.py
• Choose between locally saved models or those from our HF by toggling load_from_hf
• Stories generated per prompt, temperature and top_k can be modified
• Various tokenizers can be used for inference
• We provide direct support for Sarvam, SUTRA and Tikoken
• For adding your own tokenizers a complete understanding of sample.py is recommended :)
• Multiple prompts (each on a new line) can be mentioned in a text file.
  • Refer to training-inference/<language>-prompts.txt
  • Model can be asked to generate multiple unique stories for each prompt.
  • .txt and .json outputs are supported.

To run inference, run the script:

python training-inference/sample.py 

📊 Evaluate Inference/Stories

• We repurpose the data generation code to send stories (using G4F) to SOTA LLMs for evaluation
• Set evaluate to True in prompting/make_requests.py and run the script
• Each of our SLMs is evaluated in the following manner:
• We prepared 1000 equivalent prompts for each language training-inference/prompt-<langugage>.txt
• These 1000 prompts cover themes such as Adventure, Creativity, Courage, etc..
• All the themes and other details can be found in prompting/o1_inference-prompt-generation
• Each model produces 3 stories per prompt which are sent to SOTA LLMs for eavluation

For each model, 3000 stories are evaluated according to the prompt:

f'''{story}

The given {language} short story is for 5-7-year-old children.
Keeping in mind the target demographic, rate the story on a scale of 1-10 for context awareness, completeness, grammar, fluency, and creativity.
Evaluate context awareness by strictly assessing how well the story's middle and end align with the prompt "{prompt}".
Also, provide an overall rating on a scale of 1-10.

Only return a JSON dictionary in the following format: 
{
"context awareness": "your_context-awareness_score",
"completeness": "your_completeness_score",
"grammar": "your_grammar_score",
"fluency": "your_fluency_score",
"creativity": "your_creativity_score",
"overall": "your_overall_rating"
}'''

📈 Results

[!NOTE]

• In line with the evaluation section above, all stories are evaluated using GPT-4o

⚙️ Hyperparameter Comparisons

• This subsection covers results for models trained on synthetic (generated by GPT-4o) data
• We recommend not to compare scores across languages

Table 1: Hindi 🪔 – Hyper-parameter Comparison

Higher the better — Scores are out of 10 — 3000 stories per configuration evaluated

Table 2: Marathi 🥁 – Hyper-parameter Comparison

Higher the better — Scores are out of 10 — 3000 stories per configuration evaluated

Table 3: Bangla 🐅 – Hyper-parameter Comparison

Higher the better — Scores are out of 10 — 3000 stories per configuration evaluated

Key Findings

• SLMs with as few as 53M parameters can consistently generate well-defined stories in Indian languages
• Increasing model size from 4.5M to 153M parameters consistently improves performance across all languages
• Grammar scores show the highest performance, while context awareness remains the most challenging
• Smaller (5M) models can generate coherent stories, albeit with poor contextual awareness
• The optimal configuration appears to be 53M: Hidden Dimension: 512 with Layers: 6, balancing performance and efficiency

💡 Inference Results

• All inference examples are from models trained on synthetic data
• Starting prompts for inference for each story are mentioned in ** ... **
• Currently, models only support story completion based on starting prompts :)

Hindi 5M

**घने जंगल में, एक चमकदार पेड़ रहस्यमय शक्ति से गूंज रहा था।** एक दिन, उन्होंने देखा कि पेड़ पर बहुत सारे रंग-बिरंगे फूल खिले हुए हैं। मोहन ने सोचा, "ये फूल कितने सुंदर हैं!" उसने अपने दोस्तों से कहा, "चलो, हम यहाँ कुछ खास करते हैं।" सभी दोस्त एक साथ मिलकर पेड़ के नीचे बैठ गए।

बच्चे बहुत खुश थे और उन्होंने तय किया कि वे यहाँ कुछ मजेदार करेंगे। मोहन ने एक छोटी सी बंदूक मांगी और कहा, "हम इसे पानी देंगे ताकि यह और भी सुंदर बने।" सभी ने मिलकर पत्थर को पानी दिया और बहुत ध्यान से काम किया। लेकिन एक दोस्त ने कहा, "तुम्हें इसे बहुत पसंद आएगा।" मोहन ने समझाया कि हमें एक-दूसरे के साथ मिलकर काम करना चाहिए।

फिर, मोहन ने कहा, "हम सबको मिलकर काम करना चाहिए है। हमें हमेशा मिलकर काम करना चाहिए।" सभी ने सहमति जताई और एक पेड़ के नीचे बैठ गए। जैसे ही उन्होंने घंटी को बजाया, पेड़ ने एक मीठी आवाज़ में कहा, "धन्यवाद, बच्चों! तुमने मेरे लिए यह कर दिया।" सभी बच्चे चौंक गए और पेड़ के पास बैठ गए। उस दिन मोहन और उसके दोस्तों ने सीखा कि एकता में ताकत होती है। उन्होंने समझा कि जब हम मिलकर काम करते हैं, तो हम हमेशा एक-दूसरे की मदद कर सकते हैं। और इस तरह, मोहन और उसके दोस्तों ने एक नई दोस्ती की शुरुआत की।

>> Evaluation: Context: 6 | Completeness: 8 | Grammar: 9 | Fluency: 8 | Creativity: 7 | Overall: 7.5

**In the dense forest, a glowing tree was echoing with mysterious power.** One day, they saw that many colorful flowers had bloomed on the tree. Mohan thought, "These flowers are so beautiful!" He said to his friends, "Come on, let's do something special here." All the friends sat together under the tree.

The children were very happy and they decided that they would do something fun here. Mohan asked for a small gun and said, "We will give it water so that it becomes even more beautiful." Everyone gave water to the stone together and worked very carefully. But one friend said, "You will really like this." Mohan explained that we should all work together.

Then Mohan said, "We all should work together. We should always work together." Everyone agreed and sat under a tree. As soon as they rang the bell, the tree said in a sweet voice, "Thank you, children! You did this for me." All the children were surprised and sat near the tree.

That day, Mohan and his friends learned that there is strength in unity. They understood that when we work together, we can always help each other. And in this way, Mohan and his friends began a new friendship.

Hindi 53M

**घने जंगल में, एक चमकदार पेड़ रहस्यमय शक्ति से गूंज रहा था।** पेड़ की पत्तियाँ सूरज की रोशनी में चमक रही थीं। बच्चे उस पेड़ के पास गए और देखा कि पेड़ के नीचे एक प्यारा सा खरगोश बैठा है। खरगोश ने कहा, "मैंने सुना है कि इस जंगल में एक जादुई फूल है, जो हर साल एक बार खिलता है, तुम्हारी इच्छा पूरी करता है।"

बच्चों ने सोचा कि अगर वे उस जादुई फूल को ढूंढ लेंगे, तो उनकी सभी इच्छाएँ पूरी हो जाएँगी। उन्होंने खरगोश से पूछा, "हम उस फूल को कैसे ढूंढ सकते हैं?" खरगोश ने मुस्कुराते हुए कहा, "तुम्हें अपने दिल की सच्चाई से काम करना होगा।"

बच्चों ने मिलकर जंगल में खोज शुरू की। उन्होंने कई पेड़ और झाड़ियाँ पार कीं। अंत में, उन्होंने एक चमकदार पेड़ देखा, जो पहले से भी ज्यादा सुंदर था। बच्चों ने उस पेड़ के चारों ओर बैठकर अपनी इच्छाएँ मांगी। अचानक, पेड़ पर एक सुगंधित फूल खिलता हुआ दिखाई दिया। बच्चों ने उस फूल को देखकर एक-दूसरे को देखा और उनके दिलों में एक रहस्यमय रूप से गरमाहट भरा अनुभव हुआ।

>> Evaluation: Context: 9 | Completeness: 9 | Grammar: 9 | Fluency: 9 | Creativity: 8 | Overall: 8.8

**In the dense forest, a glowing tree was echoing with mysterious power.** The leaves of the tree were shining in the sunlight. The children went near that tree and saw that a cute little rabbit was sitting under the tree. The rabbit said, "I have heard that there is a magical flower in this forest, which blooms only once a year and fulfills your wish."

The children thought that if they found that magical flower, all their wishes would come true. They asked the rabbit, "How can we find that flower?" The rabbit smiled and said, "You must work with the truth of your heart."

The children together began searching in the forest. They crossed many trees and bushes. Finally, they saw a glowing tree, even more beautiful than before. The children sat around that tree and made their wishes. Suddenly, a fragrant flower bloomed on the tree. Seeing that flower, the children looked at each other and felt a mysteriously warm sensation in their hearts.

Marathi 5M

**सूर्योदयाच्या वेळी, दोन घट्ट मित्र नदीकाठच्या अनोळखी पाऊलखुणांचा पाठलाग करत होते.** त्यांनी एकत्र येऊन एक मोठा झाड पाहिला, जो त्यांना पाहून त्यांच्यातील आवाज ऐश्वर्यपूर्ण आवाजात एक अद्भुत दृश्य दिसलं. ते झाड खूप सुंदर होते. आर्यन त्या झाडाच्या पानांवर चढून खेळत होता.

त्यांच्या आवाज आणि आर्यनने ठरवलं की, ते झाडावर चढून, झाडावर चढतील. झाडावर चढल्यानंतर आर्यनने एका मोठ्या झाडाच्या खाली एक मोठा पक्षी पाहिला. तो पक्षी आकाशात उडत होता, पण आर्यन खूप खूष झाला. आर्यनने त्या पक्षाला स्पर्श केला आणि त्याच्या पानावर एक मोठा आवाज आला.

त्या दिवशी आर्यनने त्या पक्ष्याने एक जादुई पंख घेतला आणि त्या पक्ष्याला एकत्र बसले. ते पक्षी आकाशात उडायला लागले, पण चिरंतन एकदम सुंदर होता. तो पक्षी खूप सुंदर आणि सुंदर होता, तो त्याला म्हणाला, "तू एकटा असणारा गाणं गाणं ऐकतो, परंतु मी तुझ्या सोबत आहेस!" आर्यनने हळू हळू हळूच उचलून त्याच्या गाण्यात रमले एक अद्भुत गोष्ट शिकली. आता तो एकदम शांत होता, जो सर्वांच्या मनात राहिला.

>> Evaluation: Context: 6 | Completeness: 7 | Grammar: 6 | Fluency: 6 | Creativity: 8 | Overall: 6.5

**At sunrise, two close friends were following some unfamiliar footprints by the riverside.** Together, they came upon a large tree, and as they looked at it, a majestic voice echoed within them, revealing a wondrous sight. The tree was incredibly beautiful. Aryan began climbing and playing among its leaves.

Moved by the voice and their curiosity, Aryan decided to climb the tree. Once at the top, he saw a large bird resting beneath the tree. The bird soon soared into the sky, and Aryan felt immense joy. As he reached out and touched the bird, a loud sound came from the leaves.

That day, Aryan received a magical feather from the bird, and he sat with the bird. The bird began flying through the sky, and the moment was breathtaking. The bird was stunning and graceful. It said to Aryan, “You may sing songs alone, but I am with you!” Slowly, Aryan lifted his spirits and became immersed in its song, learning a wonderful lesson. From that moment on, he was filled with peace—a peace that stayed in everyone’s hearts.

Marathi 53M

**सूर्योदयाच्या वेळी, दोन घट्ट मित्र नदीकाठच्या अनोळखी पाऊलखुणांचा पाठलाग करत होते.**

त्यांनी एक गडद जागा पाहिली जिथे एक मोठा, चमकणारा दगड होता. "हे काय आहे?" समीर विचारला. आर्यनने उत्तर दिलं, "कदाचित हे नक्षत्र आहे. चला, आपण त्या दगडाच्या मागे जाऊन पाहू!" ते धाडसाने दगड ओलांडू लागले. दगडाच्या मागे एक लपलेलं गुहा दिसलं.

गुहेच्या आत गेल्यावर, त्यांना अनेक रंगीत रत्न आणि चमकदार वस्त्रं सापडली. समीर आणि आर्यन खूप खूश झाले. त्यांनी ठरवलं की ते या गुप्त ठिकाणाला परत येतील. त्या दिवशी, त्यांनी खूप मजा केली आणि गुप्त ठिकाण शोधून काढले. त्यांनी एकत्र येऊन त्या गुप्त ठिकाणाबद्दल आपल्या मित्रांना सांगितलं. त्या दिवशी त्यांनी एक अद्भुत अनुभव घेतला, जो सदैव लक्षात राहील.

>> Evaluation: Context: 7 | Completeness: 9 | Grammar: 8 | Fluency: 8 | Creativity: 8 | Overall: 8

**At sunrise, two close friends were following unfamiliar footprints along the riverside.**

They saw a dark place where there was a large, glowing stone. "What is this?" Sameer asked. Aryan replied, "Maybe it's a star fragment. Come on, let’s go see behind that stone!" They bravely started crossing the stone. Behind the stone, they saw a hidden cave.

When they entered the cave, they found many colorful gems and shining robes. Sameer and Aryan were very happy. They decided they would return to this secret place. That day, they had a lot of fun and discovered a hidden spot. Together, they told their friends about the secret place. That day, they had a wonderful experience that they would always remember.

Bangla 5M

**সূর্যোদয়ের সময়, দুই সেরা বন্ধু নদীর ধারে অদ্ভুত পায়ের ছাপ অনুসরণ করছিল।** সে মনে মনে ভাবছিল, আজকের দিনটি কত সুন্দর হয়েছে। সে দৌড়াতে দৌড়াতে নদীর কাছে পৌঁছাল। নদীর জল ছিল পরিষ্কার এবং নীলু ফুলটি ঝরঝরে।

সাগরের নিচে পৌঁছানোর পর, রবি দেখতে পেল একটি ছোট্ট মাছ। মাছটি খুব সুন্দর ছিল এবং তার রঙ ছিল সাদা। রবি মাছটিকে খুব ভালোবাসত। সে মাছটিকে কাছে নিয়ে গেল এবং কিছুক্ষণ মাছটিকে দেখল। মাছটি বলল, "তুমি খুব ভালো কাজ করেছ! আমি তোমার জন্য জল ছিটিয়ে দিতে চাই।" রবি খুব খুশি হলো এবং মাছটিকে আরও কাছে নিয়ে গেল।

রবি মাছটিকে ধন্যবাদ জানাল। সে জানাল, মাছটি তার পাশে আছে, যেন তার সাথে খেলা করছে। রবি মাছটির সাথে খেলা শুরু করল এবং মাছটি তার সাথে দৌড়াতে লাগল। সেদিন রবি বুঝতে পারল, প্রকৃতির সাথে বন্ধুত্ব করা সবসময় ভালো।

>> Evaluation: Context: 6 | Completeness: 7 | Grammar: 8 | Fluency: 7 | Creativity: 8 | Overall: 7.5

**At sunrise, two best friends were following strange footprints by the river.** One of them was thinking to himself how beautiful the day had become. He ran and reached the riverside. The river’s water was clear, and the blue flower was fresh.

After reaching under the sea, Robi saw a little fish. The fish was very beautiful and its color was white. Robi loved the fish very much. He brought the fish close and watched it for a while. The fish said, "You’ve done a very good job! I want to sprinkle water for you." Robi was very happy and brought the fish even closer.

Robi thanked the fish. He felt like the fish was beside him, as if playing with him. Robi started playing with the fish, and the fish began to run along with him. That day, Robi realized that making friends with nature is always a good thing.

Bangla 53M

**সূর্যোদয়ের সময়, দুই সেরা বন্ধু নদীর ধারে অদ্ভুত পায়ের ছাপ অনুসরণ করছিল।** তাদের কাছে এক অদ্ভুত দারুন ছিল, তারা যেন তাদের জন্য নতুন একটি গল্প বলবে। এই গল্পটা ছিল খুব মজার এবং তারা সেই গল্প বলার সময় হাসাহাসি করতে লাগল।\

সন্ধ্যা হলে, তারা বাড়ির পথে হাঁটতে থাকল। সূর্য তখন আকাশে লাল হয়ে উঠেছিল এবং নদীর পাড়ে আধো আলোয় সবকিছুকে উজ্জ্বল করে দিয়েছিল। শিশুরা একসাথে হাঁটতে হাঁটতে মনে মনে ভাবছিল, আজকের দিনটা কত সুন্দর ছিল! তারা জানত, এই ধরনের সময়গুলো তাদের বন্ধুত্বকে আরও শক্তিশালী হবে।

এখন তারা প্রতিদিন খেলতে আসবে, নতুন নতুন গল্প বলবে। নদীর শান্ত জল এবং সূর্যের আলোতে তাদের খেলা সব সময় মনে রাখার মতো থাকবে।

>> Evaluation: Context: 9 | Completeness: 9 | Grammar: 8 | Fluency: 9 | Creativity: 8 | Overall: 9

**At sunrise, two best friends were following strange footprints by the riverbank.** They had a strange charm with them, as if it would tell them a new story. That story was very funny and while telling it, they started laughing.

In the evening, they began walking home. The sun had turned red in the sky and lit everything brightly along the riverbank in the dim light. The children, walking together, were thinking to themselves how beautiful the day was! They knew that moments like these would make their friendship even stronger.

Now they will come to play every day, tell new stories. Their games in the calm water of the river and in the sunlight will always be something to remember.

🆚 Synthetic vs Translated

• We compare evaluation scores of the 53M (Sarvam) models trained on synthetic data (generated by 4o) vs translated data (deeptranslate)

Key Findings

• Training solely on our carefully generated synthetic corpus significantly boosts model performance across all metrics, across the evaluated languages
• We conclude that dataset generation, for SLM/LLM training using current translation tools, falls far short of synthetic data generation via SOTA LLMs.

🆚 Tokenizer Comparison

We compare models trained on our synthetic data tokenized via various Indic and non-Indic tokenizers

Table 4: Comparison of Tokenizers across Hindi, Marathi, and Bangla

Higher the better — Scores are out of 10 — 3000 stories per configuration evaluated

🪔 Hindi (53M Model)

🥁 Marathi (53M Model)

🐅 Bangla (53M Model)

Key Findings

• Language-specific tokenizers (Sarvam, SUTRA) consistently outperform general-purpose alternatives (Tiktoken)
• Sarvam achieves the highest overall scores for Hindi and Bangla, with competitive performance for Marathi
• Tiktoken shows the lowest perplexity but struggles with capturing context and completeness
• Renyi entropy analysis shows Sarvam produces more concentrated token distributions (lower entropy values) compared to SUTRA
• MorphScore analysis reveals SUTRA better preserves morphological boundaries despite higher entropy

🔎 Language Complexity

[!NOTE]

• We employ a dual-perspective approach using MorphScore and Rényi entropy to quantitatively analyze the linguistic complexity of three major Indian languages: Hindi, Bengali, and Marathi
• We compare language complexity given a certain tokenizer. Results may vary for different tokenizers.
• Language complexity refers to how hard it is for the SLM to "learn" the training data.

MorphScore

• We analysed the morphological alignment of tokenizers for each language using the MorphScore, which quantifies alignment between tokenizer outputs and linguistic morphemes.
• A morpheme is the smallest unit of language with meaning, serving as a basic building block for words.

Table 5: MorphScore Analysis

Higher MorphScore indicates better alignment with true morpheme boundaries

Rényi Entropy

• To assess the tokenization quality of Sarvam and SUTRA for Indic languages, we employed Rényi entropy as an information-theoretic metric to quantify the uncertainty and diversity in the token distribution resulting from each tokenizer.
• Specifically, we computed Rényi entropy with 𝛼 = 2.5 across the training corpora for Hindi, Bengali, and Marathi using both tokenizers.
• The choice of 𝛼 emphasizes high-probability tokens, highlighting token concentration and how evenly probability mass is distributed across the vocabulary.y.

Table 6: Rényi Entropy Analysis

• Higher Rényi entropy is better when diverse, balanced token usage is desired
• Lower Rényi entropy is better when concentrated; efficient tokenization on frequent tokens is preferred.

✅ A Fitting Use Case

For Researchers

• Comparative analysis of tokenizers, translation methods, and language complexity is resource-intensive with LLMs due to high training and inference costs.
• We introduce a lightweight framework using Small Language Models (SLMs) to enable efficient, scalable comparisons—offering results aligned with LLMs at a fraction of the cost.

General Purpose

• Regional languages are underrepresented in NLP research.
• Our SLM framework helps bridge this gap by enabling fast, low-cost generation of regional short stories.
• This supports educators in creating child-friendly content with minimal effort.

We would love to help you!

• Please contact us if you wish to learn more about training SLMs for your unique regional languages!

💰 Costs

[!IMPORTANT]

• Pipeline Overview:
  • Prompt Generation (free 🎁)
  • Data Generation (free using G4F 🎁)
  • Training a SLM (<20 USD using TensorDock 💚)
  • Inference (CPU inference supported ~free 🎁)
• Total Cost to generate your custom Regional-SLM ~15 USD :)
• First Time Setup Effort:
  • Assuming intermediate competancy with DL and LLMs
  • 2-6 hours; Time is money, after all :)

📋 Training Regime

• We utilise a DDP (Distributed Data Parallel) setup for multi-GPU training.
• Each GPU randomly samples a batch from the training/testing data.
• Total training time is only affected by the total combined VRAM (across all GPUs).

• Training Epochs: 5000 (until convergence)
• Testing Epochs: 50
• Testing Frequency: Every 200 epochs
• Logging Frequency: Every 2 epochs

⚙️ Hardware Details & Cost

• Using 2×H100 doubles both VRAM and hourly cost but halves the training time, keeping the total cost unchanged.
• As training is VRAM-bound rather than FLOP or architecture-dependent, the RTX A6000 offers the best cost efficiency per GB of VRAM.

📝 Citation

If you use Vizuara's TinyStories Regional in your research, please cite us using the following BibText template:

@misc{patil2025regionaltinystoriesusing,
      title={Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performance}, 
      author={Nirvan Patil and Malhar Abhay Inamdar and Agnivo Gosai and Guruprasad Pathak and Anish Joshi and Aryan Sagavekar and Anish Joshirao and Raj Dandekar and Rajat Dandekar and Sreedath Panat},
      year={2025},
      eprint={2504.07989},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2504.07989}, 
}