LLaMA-Omni for Basque

Speech-to-text fine-tuning of LLaMA-Omni style speech-language models for Basque (euskara), with a parallel set of Maltese experiments.

The work compares three speech-language model configurations on Basque ASR:

Training is full fine-tuning with the speech encoder frozen (--freeze_parameters model.speech_encoder), swept over learning rate (5e-6 … 1e-4) and effective batch size (32 … 256).

---

Relationship to upstream projects

This repository is a fork of ms-swift v2.6.1 (Apache License 2.0). The fork adds Basque/Maltese speech-language model support to the SWIFT training framework; the changes are confined to four files:

• swift/llm/utils/model.py — registers the llama3_1-8b-omni, llama3_1-8b-omni-latxa and llama3_1-8b-omni-latxa-skeleton model types
• swift/llm/utils/template.py — adds the corresponding prompt templates
• swift/llm/utils/argument.py, swift/llm/infer.py — supporting changes

Model code is pulled from a fork of LLaMA-Omni (ictnlp/LLaMA-Omni) as a git submodule.

The original upstream README is preserved at docs/upstream/README-llama-omni.md. See LICENSE for the Apache 2.0 terms inherited from ms-swift.

---

Setup

git clone --recursive <this-repo>
cd llama-omni-basque
bash install.sh

install.sh creates the environment and clones the LLaMA-Omni submodule. It pins pip==24.0, setuptools==69.5.1, deepspeed==0.15.4 and transformers==4.44 — these versions matter, newer ones break the LLaMA-Omni model code.

Configuration

All machine-specific paths live in a single gitignored file. Copy the template and edit it:

cp env.sh.example env.sh
$EDITOR env.sh

env.sh defines where the repo, the corpora and the checkpoints live, and an activate_env() function that puts the swift CLI on your PATH. Every training and inference script sources it, so this is the only file you need to change to run the project on a different machine:

The Slurm scripts keep their #SBATCH --partition / --account / --constraint directives as used on the original cluster — edit these to match your site, since Slurm parses them before any shell code runs and they cannot be driven from env.sh.

---

Running

Submit from the repository root (the scripts locate env.sh via $SLURM_SUBMIT_DIR):

# Basque, LLaMA-Omni
sbatch euskara/scripts/train.slurm
sbatch euskara/scripts/infer.slurm

# Basque, Whisper-Omni
sbatch euskara/scripts/whisper_omni/train.slurm

# Maltese
sbatch maltese/experiments/llama/scripts/train/mt_lr1e-4bs32ep6.slurm

get_save_steps.sh provides get_save_steps() and validate_batch_size(), used by the training scripts to derive checkpoint/eval intervals from the sample count and to assert that num_gpus × batch_per_gpu × grad_acc == effective_batch_size.

Interactive demo

inference/app.py is a Gradio demo that transcribes uploaded audio with either fine-tuned model. It reads checkpoint locations from CKPT_ROOT:

source env.sh && python inference/app.py

Standalone Whisper baseline

maltese/whisper/ is a self-contained Whisper fine-tuning baseline, independent of SWIFT. It is configured through maltese/whisper/config.yml with CLI overrides:

python maltese/whisper/train.py --config maltese/whisper/config.yml

---

Repository layout

euskara/scripts/       Basque training + inference Slurm scripts
maltese/               Maltese experiments and standalone Whisper baseline
inference/             Gradio demo (app.py) and API notebooks
experiments/           Run configs and training logs (see below)
notebook/              Analysis and plotting notebooks
swift/                 Forked ms-swift framework
docs/upstream/         Preserved upstream README

Experiment logs

experiments/ retains the run configurations (sft_args.json, training_args.json) and per-step training logs (logging.jsonl) for the learning-rate / batch-size sweeps under experiments/logs/all_llama/ and experiments/logs/all_latxa/. notebook/plots.ipynb reads these to produce the loss and WER figures.

Absolute paths in these artifacts have been replaced with ${WORK_DIR} and ${DATA_ROOT} placeholders. TensorBoard event files and raw inference dumps are not included.

---

Data

Basque training uses composite_corpus_eu_v2.1, which aggregates Common Voice, OpenSLR and Basque Parliament speech. Maltese uses the MASRI corpora together with Common Voice Maltese.

The corpora are not redistributed here — obtain them from their original sources under their respective licenses and point DATA_ROOT at them.

Results over Faktoria radio broadcast material appear in the thesis but are not included in this repository, pending confirmation of redistribution terms.

---

Citation

If you use this work, please also cite the upstream projects it builds on: LLaMA-Omni and ms-swift.