Vocal Music under Phoneme-Conditional Analysis

Code and materials for the ISMIR 2026 paper Vocal Music under Phoneme-Conditional Analysis — Hayoon Kim and Kyogu Lee, Music and Audio Research Group, Seoul National University.

The vocal music of each language carries a distinctive sonic identity, even without instrumental accompaniment. We ask whether these differences are measurable and traceable to specific phonemes. We introduce phoneme-conditional analysis, which isolates the acoustic effect of typologically distinctive phonemes by comparing marker syllables against matched non-marker controls within the same song, holding singer, melody, and genre constant.

Across nine typologically diverse languages and thousands of songs, we measure effects along five acoustic dimensions. Song-level profiles built from these effects identify the language of an unaccompanied vocal at 85.5% balanced accuracy in a nine-way classification with folds grouped by artist (chance = 11.1%).

What the method does

For every phoneme the pipeline finds a matched control in the same song — the same phone type, as close in time as possible — so that a marker/non-marker comparison holds the performer, the melody, and the production constant. The contrast is therefore within-song, not between-corpus.

Effects are measured in five modules: articulatory impact (formants, spectral tilt, H1–H2, HNR), melodic deviation (F0), rhythmic adaptation (duration), melisma (notes per phoneme), and timbral signature (centroid, MFCC).

Headline results

Preservation from speech to singing is channel-dependent: formant coarticulation survives where the constriction is a tongue-body gesture the singer must make anyway (Mandarin retroflex F2 fully preserved), while duration is uniformly attenuated and in two cases reverses sign.

Layout

pipeline/          the processing pipeline
  scaffold/
    phase2/          separation, F0, alignment, lyrics→IPA, marker
                     selection, feature extraction, quality
    rmvpe/           RMVPE F0 estimator
    lid.py           language identification
    fingerprint.py   Chromaprint dedup
    quality_gate.py  ingestion quality gate
  scripts/
    align_v3_batch.py  the original aligner (ASR-anchored)
    align_v4.py        one CTC pass over the whole track, MMS romanized
    align_v5.py        IPA-native model + gated spectral-flux snapping
    align_final.py     blend of the two, 0.6 towards the IPA model
    extract_features_v3.py
    run_analysis_v3_v5_grouped.py   effect sizes + classification
    run_lmm_v3.py                   linear mixed models
    make_all_figures_v5.py
  configs/           phase-2 processing configs
  run_phase2.py      pipeline driver

benchmark/         alignment benchmark — see benchmark/README.md
  prepare_nus.py, prepare_gtsinger.py   build the benchmark sets
  score.py, score_gtsinger.py           metrics
  run_mfa.py                            Montreal Forced Aligner comparison
  finetune_sing.py                      fine-tune the IPA CTC model on singing
  sweep_*.py                            option sweeps, singer-level CV
  rebuild_features.py                   recompute features from a new alignment
  recompute_table.py                    Table 4, speech vs. singing
  check_artist_leakage.py               artist overlap across folds
  duration_error_propagation.py         what alignment error does to the effects
  plot_*.py, redraw_confusion.py        paper figures
  emit_latex_tables.py, vizstyle.py     paper tables, shared plot style
  test_reproduction.py                  asserts all of the above against
                                        the paper's reported numbers

results/           derived results behind the paper
  cohens_d.csv                    paired Cohen's d, language × module (Table 2)
  lmm_results.csv                 LMM coefficients, language × module × feature
  lmm_results_v3_alignment.csv    the same on the original alignment, for the
                                  before/after heatmap
  classification_cache.npz        the 35-d song-level vectors, labels, and
                                  artist groups the Random Forest is fit on
  phoneme_property.csv            Table 3's cells: one d per property × feature
  song_level_cache.pkl            per-song nPVI and melisma rate
  keep_tracks.json                the 5,024 track IDs that pass the
                                  alignment-agreement gate — the analysed set
  tables/                         the paper's LaTeX tables

figures/           the paper's figures

Install

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

Two system dependencies are not pip-installable:

• espeak-ng for G2P — apt install espeak-ng
• Montreal Forced Aligner, only for the comparison in benchmark/run_mfa.py — install via conda, see the MFA docs

The two acoustic models behind the alignment are fetched on first use: torchaudio.pipelines.MMS_FA, which ships with torchaudio, and facebook/wav2vec2-lv-60-espeak-cv-ft from HuggingFace. The fine-tuned variant of the second is not published; MODELS_FT points at a local copy, and benchmark/finetune_sing.py retrains it from GTSinger.

Reproducing

Without the corpus

results/ ships the derived statistics the paper's tables and figures are built from, so four of them regenerate from a fresh clone. These need only

pip install numpy scipy matplotlib scikit-learn

— not the full requirements.txt, since no audio is touched. Each command below was run against a clean clone of this repository; the right-hand column is what it prints.

python benchmark/emit_latex_tables.py        # Tables 2 and 3
python benchmark/redraw_confusion.py         # Figure 3, confusion matrix
python benchmark/plot_song_level.py          # Figure 2, nPVI and melisma rate
python benchmark/ablate_features.py          # the block ablation in §3.6
python benchmark/plot_fingerprint_heatmap.py # supporting: d heatmap, before/after
python benchmark/plot_mandarin_at.py         # supporting: Mandarin A vs T

redraw_confusion.py and ablate_features.py refit the Random Forest with artist-grouped 5-fold CV from results/classification_cache.npz, so those accuracies are recomputed rather than quoted.

To run all six at once and assert every number above against the paper:

python benchmark/test_reproduction.py

That is what CI runs on each push, along with a check that no absolute path from the authoring machine has crept back in.

Table 3 is built from results/phoneme_property.csv, which holds one Cohen's d per cell — all the table needs, where the per-pair cache behind those cells is 85 MB. Set PHONPROP_CACHE to recompute the cells from such a cache instead; benchmark/plot_phoneme_property.py builds one from the corpus. Note that §4.1's counts of cells clearing 0.10, 0.15 and 0.20 (21, 11 and 6) are counted on the two-decimal values the table prints, not on the raw d.

Table 4 (speech vs. singing) is the one table that still needs the corpus: benchmark/recompute_table.py reads the aligned tracks.

Where each result in the paper comes from

figures/ holds the paper's two data figures (song_level, confusion_matrix) plus four that support claims made in the prose rather than appearing as numbered figures: the before/after d heatmaps, the Korean and Mandarin breakdowns, and the alignment-validation panel.

From audio

Requires the corpus (see below). The stages run in this order — note that a new alignment means the features must be rebuilt before any analysis, since the LMM draws from two feature files that are both functions of the phone boundaries:

cd pipeline
python run_phase2.py --stage separation             # Mel-Band RoFormer vocals
python run_phase2.py --stage lyrics_ipa             # lyrics → IPA, per-language G2P
python run_phase2.py --stage f0                     # FCPE + RMVPE ensemble
python scripts/align_final.py                       # the paper's alignment (v4+v5 blend)
cd ../benchmark
python rebuild_features.py --tg alignment_v5.TextGrid --out-v3 features_v5.npz
cd ../pipeline
python scripts/run_lmm_v3.py                        # mixed models → lmm_results.csv
python scripts/run_analysis_v3_v5_grouped.py        # effect sizes, classification
python scripts/make_all_figures_v5.py               # figures

Run the pipeline/ commands from pipeline/, which is where scaffold is importable from.

The stages are listed separately on purpose: do not run run_phase2.py --stage alignment, and do not run it with the default --stage all, which includes that stage. It is an earlier encoder-based aligner, superseded by align_final.py, and it imports clap.encoders from anyspeech, which is not on PyPI — so it cannot run here anyway. The paper's alignment is the align_final.py step above; rebuild_features.py writes it to alignment.TextGrid, the filename the rest of the phase-2 stack reads.

Paths. No script contains a path from the machine this work was done on. Anything inside the repository is found relative to the script being run; everything outside it comes from the environment, with a /path/to/... placeholder as the default so a missing setting fails loudly:

The offline commands need none of these.

Data availability

The corpus audio is not redistributed. It is 96 GB of commercial recordings sampled from Spotify national daily charts and matched to YouTube audio streams; we hold no right to redistribute it, and the paper's analyses use only metadata and derived feature vectors.

The corpus-acquisition scripts are also not included here. They drive bulk YouTube downloads through rotating residential proxies and rotating account cookies; that machinery is specific to our collection run and not needed to reproduce any result in the paper. Section 3.2 of the paper describes the procedure — chart sampling, four-signal language identification, Chromaprint deduplication — in enough detail to reimplement.

The benchmark corpora come from their original authors:

• NUS-48E — Duan, Fang, Li, Sim and Wang, The NUS sung and spoken lyrics corpus: a quantitative comparison of singing and speech, APSIPA 2013. Request it from the Sound and Music Computing Lab at NUS.
• GTSinger — Zhang et al., 2024, github.com/GTSinger/GTSinger

The alignment benchmark

The paper's alignment is validated against manual phone boundaries; the full writeup, including everything that was tried and rejected, is in benchmark/README.md. Briefly, on NUS-48E (47 sung recordings, 25,157 phones):

Given the same phone sequences, the Montreal Forced Aligner produced a usable alignment for 8 of the 47 sung recordings; on the other 39 the median error is 29–34 seconds. On read speech by the same singers the two are close.

Citation

@inproceedings{kim2026vocal,
  title     = {Vocal Music under Phoneme-Conditional Analysis},
  author    = {Kim, Hayoon and Lee, Kyogu},
  booktitle = {Proceedings of the 27th International Society for
               Music Information Retrieval Conference (ISMIR)},
  year      = {2026}
}

License

Code is released under the MIT License (see LICENSE), with one exception: pipeline/scaffold/rmvpe/ is the RMVPE pitch estimator, vendored verbatim from Dream-High/RMVPE and covered by the Apache License 2.0 that sits in that directory. See THIRD_PARTY.md.

The figures and derived result files under figures/ and results/ accompany the paper and are released under the same MIT terms. The corpus audio is neither included nor licensed here — see Data availability.