Music Captioning Pipeline

This repository contains the music captioning pipeline used for the ICME 2026 ATTM Grand Challenge. The pipeline provides two approaches for generating detailed music captions from audio files, leveraging state-of-the-art audio-language models.

Note: The official reference captions are already processed and provided at https://github.com/ntu-musicailab/ICME26-ATTM-GC-MeanAudio/tree/main/data/captions. This repository serves only as a demonstration and reference to encourage participants to explore data augmentation approaches or better leverage audio-language models (ALMs) in their solutions.

Overview

The pipeline consists of two main captioning approaches:

Pipeline A: Qwen2-Audio Direct Captioning

• Model: Qwen2-Audio-7B-Instruct
• Output: jamendo_qwen.json
• Characteristics: Generates detailed captions focusing on genre, instruments, mood, and production style without vocal-related content

Pipeline B: Music Flamingo with Post-Processing

• Models: NVIDIA Music Flamingo + Qwen3-4B-Instruct
• Output: jamendo_mf.json
• Characteristics: Generates comprehensive captions, then refines them to be more concise and removes vocal-related keywords
• Steps:
  1. Music Flamingo generates initial captions
  2. Qwen3-4B rephrases captions for conciseness and removes vocal mentions
  3. Reformatting to standard output format

Pipeline Architecture

┌─────────────────────────────────────────────────────────────┐
│                      Pipeline A                             │
│                                                             │
│  Audio Files  →  qwen_caption.py  →  jamendo_qwen.json      │
│                  (Qwen2-Audio)                              │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│                      Pipeline B                             │
│                                                             │
│  Audio Files  →  mf_caption.py  →  mf_initial.json          │
│                  (Music Flamingo)                           │
│                                                             │
│                       ↓                                     │
│                                                             │
│              qwen_rephrase.py  →  jamendo_mf_rephrased.json │
│              (Qwen3-4B)                                     │
│                                                             │
│                       ↓                                     │
│                                                             │
│            create_final_json.py  →  jamendo_mf.json         │
│            (Format standardization)                         │
└─────────────────────────────────────────────────────────────┘

Installation

Requirements

• Python 3.8+
• CUDA-capable GPU (recommended: 24GB+ VRAM for optimal batch processing)
• PyTorch 2.0+

Setup

# Clone the repository
git clone https://github.com/yourusername/ICME26-ATTM-GC-ALM-captioning-pipeline.git
cd ICME26-ATTM-GC-ALM-captioning-pipeline

Model Installation

Please refer to the original model repositories for detailed installation instructions and dependencies:

• Qwen2-Audio-7B-Instruct (for Pipeline A and general captioning):
  https://huggingface.co/Qwen/Qwen2-Audio-7B-Instruct

• Music Flamingo (for Pipeline B Step 1):
  https://github.com/NVIDIA/audio-flamingo/tree/music_flamingo

• Qwen3-4B-Instruct (for Pipeline B Step 2 - rephrasing):
  https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507

Each repository provides specific installation requirements and setup instructions.

Usage

Pipeline A: Qwen2-Audio Captioning

Generate captions using Qwen2-Audio model:

python qwen_caption.py /path/to/audio/dir output/jamendo_qwen.json

Options:

• --prompt: Custom prompt for captioning (default focuses on instrumental description)
• --batch-size: Batch size for processing (default: 6)
• --test: Test mode - process only first batch and print to terminal

Example with custom prompt:

python qwen_caption.py /data/music output.json \
    --prompt "Describe the genre, mood, and instrumentation of this track" \
    --batch-size 8

Test mode for prompt tuning:

python qwen_caption.py /data/music test_output.json --test

Pipeline B: Music Flamingo with Post-Processing

Step 1: Generate initial captions with Music Flamingo

python mf_caption.py /path/to/audio/dir output/mf_initial.json

Options:

• --prompt: Custom prompt for captioning
• --batch-size: Batch size (default: 6)
• --test: Test mode

Example:

python mf_caption.py /data/music mf_initial.json --batch-size 6

Step 2: Rephrase captions using Qwen3-4B

python qwen_rephrase.py output/mf_initial.json output/mf_rephrased.json

Options:

• --instruction: Custom rephrasing instruction
• --batch-size: Batch size (default: 8)
• --test: Test mode for instruction tuning

Example with custom instruction:

python qwen_rephrase.py mf_initial.json mf_rephrased.json \
    --instruction "Your custom rephrasing instruction here" \
    --batch-size 8

Step 3: Reformat to standard format

python create_final_json.py output/mf_rephrased.json output/jamendo_mf.json

Output Format

All scripts produce JSON files with the following standard format:

[
    {
        "path": "relative/path/to/audio.mp3",
        "caption": "Generated caption describing the music..."
    },
    ...
]

The rephrasing step (qwen_rephrase.py) produces an intermediate format that includes both original and rephrased captions, which is then cleaned up by create_final_json.py.

Features

Resume Capability

All scripts support automatic resuming from interruptions:

• Progress is saved after each batch
• Already-processed files/captions are automatically skipped
• Simply re-run the same command to continue

Test Mode

Use --test flag to process only the first batch for:

• Prompt tuning and experimentation
• Instruction optimization for rephrasing
• Quick validation before full dataset processing

Batch Processing

Configurable batch sizes to optimize for your GPU memory:

• Adjust --batch-size based on available VRAM
• Parallel data loading for efficiency
• Progress tracking with tqdm

Model Information

Tips for Data Augmentation

This pipeline can be used as a reference for data augmentation approaches:

1. Prompt Variations: Experiment with different prompts to generate diverse captions for the same audio
2. Model Ensembling: Combine outputs from both pipelines for richer descriptions
3. Rephrasing Variations: Use different rephrasing instructions to create caption variants
4. Multi-stage Processing: Apply additional filtering or refinement steps

Acknowledgments

• Qwen Team for Qwen2-Audio and Qwen3-4B models
• NVIDIA for Music Flamingo model
• HuggingFace for the transformers library