MTP-1: DeSTA2.5-Audio Large Audio-Language Model Analysis

This repository packages the MTP-1 research code for analyzing DeSTA2.5-Audio and related audio-language model behavior on MMAU and MMAR-style benchmark tasks. The public scope is intentionally narrow: DeSTA2.5-Audio backbone analysis, MMAU/MMAR evaluation, Indic and multilingual prompt transfer, decoder behavior, Q-Former and connector scaling, encoder compatibility, and baseline comparison scripts such as Qwen and Audio Flamingo where local scripts were available.

Scope

The cleaned repo focuses on:

• DeSTA2.5-Audio inference and evaluation workflows
• MMAU and MMAR benchmark handling
• multilingual and translated-prompt evaluation variants
• decoder removal and decoder-scaling analysis
• Q-Former scaling and connector behavior
• Indic Whisper / encoder swap compatibility work
• saved local result summaries with cautious wording

This repository does not present the project as tool augmentation, GRPO, RL reward training, CLE audio entailment, Audio-Maestro, or MTP-2 work.

Repository Structure

mtp1-audio-language-model-analysis/
├── README.md
├── LICENSE
├── CITATION.cff
├── requirements.txt
├── .gitignore
├── configs/
│   ├── paths.example.yaml
│   ├── inference/
│   └── evaluation/
├── docs/
├── scripts/
│   ├── inference/
│   ├── evaluation/
│   ├── baselines/
│   ├── translation/
│   └── data_preparation/
├── third_party/
│   └── desta25_audio/
├── examples/
└── tests/

Setup

Create a Python environment and install the practical baseline dependencies:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Set local path variables before running anything that expects benchmark audio or JSON files:

export MTP1_ROOT=$PWD
export MTP1_DATA_ROOT=/path/to/local/data
export MMAU_ROOT=$MTP1_DATA_ROOT/MMAU
export MMAR_ROOT=$MTP1_DATA_ROOT/MMAR
export MMAU_JSON=/path/to/mmau-test-mini.json
export MMAR_JSON=/path/to/MMAR-meta.json
export WHISPER_WEIGHTS=/path/to/local/indic-whisper
export QFORMER_WEIGHTS=/path/to/local/qformer.safetensors

Use configs/paths.example.yaml as the starting point for a private local config.

The repository prefers the vendored third_party/desta25_audio/ snapshot for reproducibility. If you choose to use a separately installed upstream copy instead, make sure it is compatible with the local scripts and configs.

Data And Checkpoint Policy

This public repo intentionally excludes:

• raw audio and video
• full benchmark JSONs and translated benchmark derivatives
• DeSTA-AQA5M dataset shards
• model weights and checkpoints
• W&B runs, logs, and generated experiment outputs
• large prediction dumps

The repository is meant to document code structure and experiment organization, not to redistribute datasets or private checkpoints.

Example Commands

Run DeSTA2.5-Audio on a local MMAU JSON:

python scripts/inference/run_desta_mmau.py \
  --mmau-root "$MMAU_ROOT" \
  --data-file "$MMAU_JSON" \
  --output "$MTP1_ROOT/outputs/predictions_mtp1_mmau.json" \
  --lang en

Run MMAR evaluation from a local metadata file:

python scripts/inference/run_desta_mmar.py \
  --mmar-root "$MMAR_ROOT" \
  --data-file "$MMAR_JSON" \
  --output "$MTP1_ROOT/outputs/predictions_mtp1_mmar.json"

Run the official-style MMAU evaluator on an already-produced prediction file:

python scripts/evaluation/mmau_official_evaluation.py \
  --input "$MTP1_ROOT/outputs/predictions_mtp1_mmau.json"

Result Summary

The repository includes only saved local prediction summaries, not certified final thesis tables. Several local result files suggest:

• an English MMAU mini DeSTA2.5-Audio result around 68.10%
• a VAD-oriented MMAU variant around 67.80%
• a no-decoder English MMAU variant around 59.40%
• multilingual MMAU variants for Hindi, French, Italian, and Kannada
• Q-Former scaling sweeps and decoder-scaling sweeps saved locally
• a local MMAR result around 49.30%
• Qwen baseline outputs around 64.00% and 65.70% on saved local files

See docs/results.md for the cautious summary and file references.

Limitations

• Many experiment scripts originated as research drivers rather than polished release utilities.
• Benchmark redistribution and translated benchmark licensing still need manual confirmation.
• Saved prediction files in the original workspace were not all clearly labeled as final thesis numbers.
• Some scripts still assume the user has local benchmark files and model access outside this repo.
• The vendored DeSTA2.5-Audio snapshot is a cleaned local dependency, but model weights and datasets still need to be obtained separately under their own terms.

Third-Party Code And License Scope

This repository vendors a cleaned DeSTA2.5-Audio source snapshot under third_party/desta25_audio/ because it is the core implementation component used in this MTP-1 work.

The upstream DeSTA2.5-Audio setup.py declares the classifier License :: OSI Approved :: MIT License; the vendored copy includes LICENSE.upstream and THIRD_PARTY_NOTICE.md for provenance.

The root MIT license applies only to this repository's original scripts, configs, docs, examples, and tests.

Datasets, benchmark files, translated derivatives, model weights, and checkpoints are not included and remain governed by their original terms.

Citation And Acknowledgement

If this repository is used in academic work, please cite the thesis or repository record described in CITATION.cff.

This repo also builds on external code and benchmark material, especially DeSTA2.5-Audio, MMAU, and MMAR-related components. Please preserve attribution and honor the original licenses and benchmark terms for any third-party material.