---

🌟 What is LyricEditBench?

LyricEditBench is the first comprehensive benchmark designed for evaluating melody-preserving lyric editing in singing voice synthesis. Built from GTSinger, it provides 7,200 carefully curated test instances covering six common lyric modification scenarios, with balanced sampling across singer gender, language, modification type, and singing technique for fair and comprehensive comparison.

✨ Key Highlights

🎯 Comprehensive Coverage

• 🎤 7,200 test instances with balanced sampling
• 🌐 2 languages: Chinese and English
• 🎭 2 genders: Male and Female
• 🎼 6 singing techniques + technique-free category from GTSinger

✏️ Six Lyric Modification Types

📏 Standardized Evaluation Protocol

• 📊 Objective metrics: PER (intelligibility), SIM (speaker similarity), F0-CORR (melody adherence), VS (vocal quality)
• 🔀 Each instance includes a melody reference, a timbre prompt, and modified lyrics

---

📊 Dataset Statistics

• 300 samples per modification type per category (gender × language)
• 30 samples per singing technique, 120 for technique-free, per combination
• Modified lyrics generated by DeepSeek V3.2, with non-compliant outputs discarded
• All melody reference clips are ≤ 15 seconds
• Timbre prompts are randomly drawn from the remaining audio pool (≤ 15 seconds)

🚀 Quick Start

Option 1: Already have GTSinger downloaded?

If you already have GTSinger locally, you can reconstruct the full dataset with local audio paths:

import json
import os
from pathlib import Path

from datasets import Audio, Dataset
from huggingface_hub import hf_hub_download


def build_dataset_from_local(gtsinger_root: str):
    """
    Build LyricEditBench dataset using your local GTSinger directory.

    Args:
        gtsinger_root: Root directory of your local GTSinger dataset.
    """
    # Download the inherited metadata from HuggingFace
    json_path = hf_hub_download(
        repo_id="ASLP-lab/LyricEditBench",
        filename="GTSinger_Inherited.json",
        repo_type="dataset",
    )

    with open(json_path, "r") as f:
        data = json.load(f)

    gtsinger_root = str(Path(gtsinger_root).resolve())

    # Prepend local root to relative paths
    for item in data:
        item["melody_ref_path"] = os.path.join(gtsinger_root, item["melody_ref_path"])
        item["timbre_ref_path"] = os.path.join(gtsinger_root, item["timbre_ref_path"])
        # Set audio fields to the resolved file paths
        item["melody_ref_audio"] = item["melody_ref_path"]
        item["timbre_ref_audio"] = item["timbre_ref_path"]

    # Build HuggingFace Dataset with Audio features
    ds = Dataset.from_list(data)
    ds = ds.cast_column("melody_ref_audio", Audio())
    ds = ds.cast_column("timbre_ref_audio", Audio())

    return ds


# Example usage
ds_full = build_dataset_from_local("/path/to/your/GTSinger")

Option 2: Use without GTSinger dependency

from datasets import load_dataset

ds_full = load_dataset("ASLP-lab/LyricEditBench", split="test")

Option 3: Download the full dataset as archive

huggingface-cli download ASLP-lab/LyricEditBench --repo-type dataset --local-dir LyricEditBench

🎯 Subset Inference

For quick evaluation, we provide subset ID lists under the id_lists/ directory (1K.txt and 3K.txt, one ID per line). You can filter the dataset by matching the id field:

from huggingface_hub import hf_hub_download


def load_subset(data: list, subset_id: str) -> list:
    """Filter dataset by a subset ID list."""
    subset_path = hf_hub_download(
        repo_id="ASLP-lab/LyricEditBench",
        filename=f"id_lists/{subset_id}.txt",
        repo_type="dataset",
    )

    with open(subset_path, "r") as f:
        id_set = set(line.strip() for line in f if line.strip())

    return [item for item in data if item["id"] in id_set]


# Example: load the 1K subset, ds_full can be obtained as shown above
subset_1k = load_subset(ds_full, "1K")
print(f"Loaded {len(subset_1k)} items")

# Example: load the 3K subset
subset_3k = load_subset(ds_full, "3K")
print(f"Loaded {len(subset_3k)} items")

🔧 Inference Scripts

See the evaluation directory in YingMusic-Singer-Plus for inference and metric computation scripts.

⚖️ License

LyricEditBench is built upon GTSinger with modified lyrics generated by DeepSeek V3.2. Following GTSinger, this dataset is released under the CC BY-NC-SA 4.0 License.

🤝 Citation

# Ours is coming soon

@article{zhang2024gtsinger,
  title={GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks},
  author={Zhang, Yu and Pan, Changhao and Guo, Wenxiang and Li, Ruiqi and Zhu, Zhiyuan and Wang, Jialei and Xu, Wenhao and Lu, Jingyu and Hong, Zhiqing and Wang, Chuxin and others},
  journal={arXiv preprint arXiv:2409.13832},
  year={2024}
}

📧 Contact & Support

🐛 Issues? Open an issue on our GitHub repository
📧 Collaboration? Contact us through GitHub
💬 Discussion? Join our Discord or WeChat Group