LFM 2.5 Audio 1.5B — ASR LoRA (Voice → Home Assistant Commands)

LoRA adapter fine-tuned on LFM2.5-Audio-1.5B to convert spoken Home Assistant commands into structured function calls (HassLightTurnOn|$area=hall).

Trained entirely on Apple Silicon using MLX. No cloud GPU required.

---

What It Does

Takes audio of a spoken command and outputs a structured function call:

Input audio: "switch on the light in the hall"
Output:      HassLightTurnOn|$area=hall

Output format: FunctionName|$arg1=val1|$arg2=val2 — pipe-delimited, parseable.

---

Training Details

---

Evaluation Results

Evaluated on Paulescu/OHF-Voice-audio-20260504 test split, 205 stratified samples (5 per function), system prompt "Perform ASR.", temperature 0.0.

The gap vs the cookbook is expected — they use full fine-tuning (all ~1.5B params) on an A100 with batch 32. This adapter trains 0.52% of parameters on a local Mac.

See also v2 — trained on the full 49,909-sample dataset, achieving 92.2% function-name and 70.0% argument accuracy with only 4,000 steps. Data volume proved more impactful than training duration.

---

Key Finding: Modality Positioning

LFM 2.5 Audio's __call__ interface concatenates audio embeddings after all text tokens, meaning the model cannot attend to audio when predicting the assistant response. For ASR fine-tuning to work, you must call model._prefill with explicit modalities so audio is positioned inside the user turn (before the assistant tokens) — matching the inference-time ChatState layout.

This is implemented in train/losses/lfm_audio_loss.py.

---

Usage

import mlx.core as mx
from mlx_audio.sts.models.lfm_audio import LFM2AudioModel, LFM2AudioProcessor
from mlx_audio.sts.models.lfm_audio.processor import ChatState
from mlx_audio.sts.models.lfm_audio.model import LFMModality

# Load base model + adapter
processor = LFM2AudioProcessor.from_pretrained("mlx-community/LFM2.5-Audio-1.5B-8bit")
model     = LFM2AudioModel.from_pretrained("mlx-community/LFM2.5-Audio-1.5B-8bit")

# Apply LoRA and load weights
from train.lora import apply_lora, load_adapters, LoRAConfig
apply_lora(model, LoRAConfig(model_type="lfm_audio", rank=16, alpha=32.0))
load_adapters(model, "path/to/adapters.safetensors")
model.eval()

# Transcribe
audio_mx = mx.array(audio_numpy)  # float32, any sample rate
chat = ChatState(processor)
chat.new_turn("system"); chat.add_text("Perform ASR."); chat.end_turn()
chat.new_turn("user");   chat.add_audio(audio_mx, sample_rate=sr); chat.end_turn()
chat.new_turn("assistant")

output = ""
for token, modality in model.generate_from_chat_state(
    chat, mode="sequential", max_new_tokens=64, temperature=0.0, top_k=1
):
    if modality == LFMModality.TEXT:
        tok_id = int(token.item())
        if tok_id == 7:  # <|im_end|>
            break
        output += processor.tokenizer.decode([tok_id])

print(output)  # e.g. HassLightTurnOn|$area=hall

Or run the included demo:

git clone https://github.com/akashicMarga/mlx-audio-train
cd mlx-audio-train
python scripts/lfm_asr_demo.py --adapter path/to/checkpoint-final
# open http://localhost:7860

---

Reproduce / Experiment

Full training code and configs are in akashicMarga/mlx-audio-train.

Things worth experimenting with:

• Higher LoRA rank (32, 64) — more capacity for memorising exact argument patterns
• Full fine-tuning — remove LoRA and train all params (needs more VRAM/RAM)
• More data — the full OHF-Voice dataset has 55K samples; we used 950
• Different system prompt — the model is sensitive to the exact system prompt used at training vs inference

# Quick 200-step smoke test (~5 min on M-series)
python scripts/train.py --config configs/lfm_audio_asr_test.yaml

# Full 10k-step run (~12 hours on M-series)
caffeinate -i python scripts/train.py --config configs/lfm_audio_asr_10k.yaml

---

Citation / Acknowledgements

• Base model: LiquidAI/LFM2.5-Audio-1.5B
• Dataset: Paulescu/OHF-Voice-audio-20260504
• Cookbook reference: Liquid4All/cookbook
• Training framework: mlx-audio