Estimin3n — Open-Source Multimodal Kazakh Audio/Text → Text LLM

Estimin3n model on Huggingface

Estimin3n: SOTA open-source multimodal Kazakh audio/text-to-text LLM

Hardware support is provided in partnership with ait🍅maton

This repository contains everything related to the Estimin3n model:

• Fine-tuning the Gemma 3N audio-language base model on custom data,
• Inference (ASR/response) using a locally saved model,
• Evaluation on audio datasets (WER/CER) and KazMMLU.

The repository is focused on Kazakh/Russian language scenarios.

Contents

• Repository structure overview
• Requirements and installation
• Data preparation
• Training (fine-tuning)
• ASR/response inference
• Benchmarks: WER/CER and KazMMLU
• Path and configuration management
• FAQ / Troubleshooting

---

Repository Structure

Estimin3n/
  bench/
    kazmmlu/
      kazmmlu.py         # KazMMLU benchmark (multiple-choice)
    wer/
      benchmark.py       # WER/CER benchmark on audio dataset
  data/
    data_save.py         # Convert corpus into internal HF DatasetDict format
  inference/
    test.py              # ASR/response inference for a single audio file
  train/
    finetune.py          # Fine-tuning Gemma 3N with Unsloth + TRL SFTTrainer
  utils/
    config.py            # Paths, defaults, utilities
    merge_lora.py        # Merge LoRA adapters with base model
    sample.wav           # Test audio file
  models/                # (auto-generated) stores final model
  outputs/               # (auto-generated) checkpoints/logs/reports
  data/datasets/         # (auto-generated) saved HF datasets
  data/raw/              # (expected raw corpus structure)

---

Requirements & Installation

Recommended versions:

• Python 3.10+
• CUDA-compatible PyTorch
• Libraries: transformers, datasets, soundfile, librosa, evaluate, jiwer, unsloth, peft, trl, tqdm, pandas

Example installation (via pip):

pip install torch --index-url https://download.pytorch.org/whl/cu121  # match your CUDA version
pip install transformers datasets soundfile librosa evaluate jiwer unsloth peft trl tqdm pandas

If using Windows, make sure soundfile (libsndfile) and librosa dependencies are installed.

---

Path & Configuration Management

Key paths and defaults are defined in utils/config.py:

• MODELS_DIR → models/
• OUTPUTS_DIR → outputs/
• DATA_DIR → data/, RAW_DATA_DIR → data/raw/, DATASETS_DIR → data/datasets/
• DEFAULT_MODEL_PATH → models/Estimin3n
• DEFAULT_DATASET_DIR → data/datasets/audio_dataset_with_text
• DEFAULT_SAMPLE_WAV → utils/sample.wav

The ensure_dirs() utility automatically creates required directories when running scripts.

---

Data Preparation

Script: data/data_save.py It processes raw data and produces an HF DatasetDict.

Expected raw corpus structure (example: Kazakh_Speech_Corpus_2/ISSAI_KSC2_formatted):

data/raw/Kazakh_Speech_Corpus_2/ISSAI_KSC2_formatted/
  train/
    **/*.flac + matching .txt transcripts
  validation/
    **/*.flac + .txt
  test/
    **/*.flac + .txt

Run conversion:

python -m data.data_save

The resulting dataset will be saved in data/datasets/audio_dataset_with_text.

---

Training (Fine-Tuning)

Script: train/finetune.py

Key points:

• Loads base Gemma 3N via unsloth.FastModel
• Builds a training dataset with system/user/assistant-style messages
• Uses trl.SFTTrainer for SFT
• Saves checkpoints and logs in outputs/; final model and processor in models/Estimin3n

Example run:

python -m train.finetune

After training, you can merge LoRA weights into the base model:

python -m utils.merge_lora

The script expects adapters in outputs/checkpoint-10000 (change path if needed inside the file).

---

Inference (ASR/Response)

Script: inference/test.py

Two usage modes:

1. Without arguments — quick test on utils/sample.wav:

python -m inference.test

2. With a custom file:

python -m inference.test \
  --audio path/to/audio.wav \
  --model-path models/Estimin3n \
  --streaming \
  --max-tokens 2048 \
  --temperature 0.8

The script automatically converts audio to 16kHz mono float32. Responses are generated by Gemma 3N using apply_chat_template and generate.

Note: Current inference/test.py includes a system prompt tailored for call-center response scenarios, not pure transcription. For ASR-only, refer to the logic in bench/wer/benchmark.py.

---

WER/CER Benchmark

Script: bench/wer/benchmark.py

Example run:

python -m bench.wer.benchmark \
  --dataset_path data/datasets/audio_dataset_with_text \
  --model-path models/Estimin3n \
  --sampling_rate 16000 \
  --output_file outputs/benchmark_log.tsv \
  --detailed_output_file outputs/benchmark_detailed.tsv \
  --max-tokens 256 \
  --temperature 0.0 \
  --show-examples \
  --show-every 10

The script:

• Loads dataset from --dataset_path (expects audio and text columns)
• Converts audio to 16kHz if needed
• Calculates intermediate and final WER/CER
• Saves logs in outputs/ and prints examples

---

KazMMLU Benchmark

Script: bench/kazmmlu/kazmmlu.py

Example run (all configs):

python -m bench.kazmmlu.kazmmlu \
  --model-path models/Estimin3n \
  --run-all \
  --max-tokens 5 \
  --temperature 0.1 \
  --show-examples \
  --show-every 50 \
  --output_file outputs/kazmlu_results.tsv \
  --detailed_output_file outputs/kazmlu_detailed.tsv

Filtering options:

• --subset "Biology (High School in kaz)"
• --kazakh-only
• --russian-only

Results are saved in outputs/ and aggregated accuracy statistics are printed.

---

Common Issues

• CUDA/torch incompatibility: install the correct PyTorch build for your CUDA version.
• soundfile/librosa errors: install system dependencies (libsndfile) and check numpy version compatibility.
• Empty responses/tokens: verify max_new_tokens, temperature, and pad_token_id in processor.
• Poor transcription: ensure prompt explicitly requests transcription (see bench/wer/benchmark.py) and audio is 16kHz mono.

---

Acknowledgements

• Google Gemma 3N
• Hugging Face transformers, datasets, trl, unsloth
• Unsloth (efficient fine-tuning in 4-bit mode)
• KazMMLU community
• tsu

Citations

If this project or its results are useful, please cite the KSC2 dataset paper:

@inproceedings{mussakhojayeva22_interspeech,
title     = {KSC2: An Industrial-Scale Open-Source Kazakh Speech Corpus},
author    = {Saida Mussakhojayeva and Yerbolat Khassanov and Huseyin {Atakan Varol}},
year      = {2022},
booktitle = {Interspeech 2022},
pages     = {1367--1371},
doi       = {10.21437/Interspeech.2022-421},
issn      = {2958-1796},
}

@article{gemma_3n_2025,
    title={Gemma 3n},
    url={https://ai.google.dev/gemma/docs/gemma-3n},
    publisher={Google DeepMind},
    author={Gemma Team},
    year={2025}
}