Awesome Audio Editing

Audio Editing in the Era of Foundation Models: A Survey

AACL-IJCNLP 2026

Changhao Pan^1,*, Yifei Fan^1,*, Fan Zhuo^1,*, Yifu Chen¹, Wenxiang Guo¹,
Yu Zhang², Ruiqi Li², Zhiyuan Zhu¹, Rui Yang¹, Shengpeng Ji³,
Chenyuhao Wen¹, Jiayang Xu¹, Ke Lei¹, Xiaoda Yang¹, Jingyu Lu¹, Zhou Zhao^1,†

¹Zhejiang University  ·  ²ByteDance  ·  ³Hunyuan Team, Tencent

^*Equal contribution  ·  ^†Corresponding author

🌐 Languages

English · 简体中文 · 한국어

🚀 Quick Start

This repository is the official repository for Audio Editing in the Era of Foundation Models: A Survey, Which is accepted by AACL-IJCNLP 2026.

• We establish a unified taxonomy of acoustic, semantic, and instance editing across speech, music, and general audio, clarifying what each task changes and what it should preserve to support consistent comparisons across editing goals.
• We review mainstream audio editing techniques through foundation-model architectures and learning paradigms, with an emphasis on their core mechanisms and suitability for different editing scenarios.
• (Updated Recently) We curate publicly available audio editing models, and summarize their supported task categories and key strengths to help readers identify suitable models.
• (Updated Recently) We organize publicly available datasets, data construction tools, evaluation benchmarks, and metrics for audio editing, summarizing the audio domains, editing categories, and evaluation dimensions they cover.

🔥What's new

• 📦 [2026/09] This repository has moved to MM-Speech/AudioEditSurvey for better management.
• 🏆 [2026/09] Our paper has been accepted to the AACL-IJCNLP 2026!
• 🎉 [2026/06] We have officially released this survey repository for Audio Editing Models, with the preprint available on arXiv.

Contents

1. Introduction
2. Scope
3. Overall
   • Taxonomy Overview
   • Taxonomy Details
   • Representative Audio Editing Methods
     • Unified · Speech · Music · Audio
4. Foundation Models for Audio Editing
5. Training-based Audio Editing
6. Training-free Audio Editing
7. Resources
   • Available Datasets
     • Speech · Music · Audio · Unified
   • Data Tools
   • Benchmarks
   • Evaluation Metrics
8. Challenges and Future Directions
9. Citation
10. Contributing

---

📌 Introduction

This is the official repository for Audio Editing in the Era of Foundation Models: A Survey, accepted to AACL-IJCNLP 2026. It is maintained by MM-Speech and collects papers and resources for foundation-model-based audio editing.

Abstract
Audio editing aims to modify a given synthetic or real-world audio signal to meet users' specific needs. As a promising yet challenging direction in AIGC, it has attracted increasing attention in recent years. With the rapid progress of text-to-audio and text-to-speech generation, powerful audio generation models have become the primary foundation for modern audio editing systems. In this survey, we provide a comprehensive review of foundation-model-based audio editing. We first define the scope of audio editing from a unified perspective and present a detailed taxonomy of existing editing tasks. We then summarize the major foundation-model paradigms for audio editing, and review representative approaches from both training-based and training-free perspectives. In addition, we systematically discuss related resources, including datasets, data construction tools, and evaluation protocols. Finally, we identify open challenges in this field and outline promising directions for future research.

---

🎯 Scope

In this survey, we focus on works that make direct contributions to audio editing in the era of foundation models. To ensure a precise and focused discussion, we adopt two main inclusion criteria: (1) the task should center on audio editing, which we define as modifying the acoustic attributes, instances, or content of an existing audio recording, without transformations so substantial that they amount to generating an entirely new audio sample; (2) the method should rely on mainstream audio foundation model paradigms. Accordingly, we do not cover works primarily focused on audio generation, nor do we provide an extensive discussion of signal-processing-based audio editing methods. In addition, to maintain a focused scope, spatial audio (multi-channel formats such as binaural stereo and first-order Ambisonics (FOA)) and related editing techniques are beyond the main scope of this survey.

---

🧭 Overall

🗂️ Taxonomy Overview

Figure 1: Taxonomy of audio editing tasks.

🧩 Taxonomy Details

📚 Representative Audio Editing Methods

Representative editors with publicly released implementations and model weights, subject to each project’s license. Editing types follow our taxonomy; Unified groups editors supporting multiple audio domains, with their supported domains listed in the table. Base links the pretrained backbone used by an editor, while Adapter links its additional learned weights.

Unified Models

Speech Models

Music Models

Audio Models

---

🏗️ Foundation Models for Audio Editing

1. Early Neural Editing Models

Before the foundation-model era, early neural audio editing methods mainly explored task-specific generative models for local reconstruction and attribute control.

2. Token-based Codec Language Models

Token-based codec language models cast audio editing as conditional generation over discrete audio tokens. After continuous audio is converted into compact discrete token sequences, target regions are edited through autoregressive continuation, infilling, or selective regeneration conditioned on context, prompts, or task controls.

3. Diffusion and Flow-Matching Models

Diffusion and flow-matching models formulate audio editing as conditional transformation in continuous acoustic spaces, such as mel-spectrograms or audio latents. Instead of infilling discrete tokens, they modify audio through conditional denoising, latent inversion, or continuous flow transformation, making them suitable for high-fidelity reconstruction, region-level refinement, and fine-grained acoustic control in complex scenarios.

4. Audio Editing Interfaces

Instruction-conditioned and multimodal interfaces for audio editing provide high-level control for foundation-model-based audio editing. They allow users to specify editing intents through natural language instructions, task prompts, reference audio, temporal regions, or visual cues, which are shifted into target spans, task embeddings, event locations, speaker references, or preservation constraints.

---

🧪 Training-based Audio Editing

Training-based approaches refer to audio editing methods that learn editing behaviors from supervised pairs, pseudo-pairs, or instruction-based triplets before inference. These methods explicitly optimize editing objectives, condition following, and preservation constraints, enabling stable and controllable editing. We group existing works into three categories based on their supervision and conditioning mechanisms, and discuss their core methods and functional scopes.

Figure 2: Overview of training-based audio editing methods.

---

🪄 Training-free Audio Editing

Training-free approaches adapt pretrained audio generative models to editing without parameter updates. They operate by manipulating inference-time mechanisms, such as inversion, attention control, prompt or guidance adjustment, and mask-based constraints. We group existing methods into three common categories, which are often combined to improve localization, preservation, and controllability. Since token-based autoregressive models are less naturally suited to training-free editing, this section mainly focuses on non-autoregressive paradigms, especially diffusion-based foundation models.

Figure 3: Overview of training-free audio editing methods.

---

📦 Resources

📊 Available Datasets

Public datasets for audio editing and controllable audio generation, grouped by their primary audio domain.

This non-exhaustive list highlights datasets suited to audio editing or widely used in the community, with availability verified by the repository maintainers for every entry.

Paired indicates released source–target audio, mixture–stem correspondence, or explicitly matched control/technique takes (✅ / ❌); shared transcripts, audio–text alignment, or audio–MIDI alignment alone do not count. † marks an editing use that requires task construction or adaptation, rather than native editing supervision. Editing types follow our Acoustic / Instance / Semantic taxonomy.

Durations are approximate, without adding together alternate modalities or mixture stems. Text refers to transcripts, captions or instructions; label-only metadata are described in Annotation.

Speech

Music

Audio

Unified

These corpora combine speech, music, and general sounds.

🛠️ Data Tools

Open-source tools for constructing editing data and annotating existing recordings. Supported Task Type follows our Acoustic / Semantic / Instance taxonomy and indicates the editing supervision that each tool can help construct. Unified covers tools applicable across speech, music and general audio.

Tools for Data Generation

Synthesis, source separation, mixing and signal processing for constructing audio examples and source–target pairs.

Speech

Music

Audio

Unified

Tools for Data Annotation

Tools for extracting or creating content, attribute and temporal annotations from existing audio.

Speech

Music

Audio

Unified

🧪 Benchmarks

Public evaluation resources for audio editing. Editing categories follow this survey's taxonomy: Acoustic / Semantic / Instance / Composite, where Composite combines different categories within one request. Evaluation method describes the scorer: Expert models, MLLM, or Hybrid (including agent-based evaluation).

MMAE

• TL;DR: Tests instruction following and preservation across speech, music, sound, and their mixtures with 2,000 cases (≈8.0 h), six complexity levels, and 17,741 verification rubrics.
• Paper: arXiv; Code: GitHub; Dataset: Hugging Face.
• Audio modalities: Speech; Music; Audio.
• Editing categories: Acoustic; Semantic; Instance; Composite.
• Evaluation method: MLLM — Qwen3-Omni judges individual rubrics with majority voting, yielding Instruction Following Rate (IFR), Consistency Rate (CR), and Exact Match Rate (EMR).
• Best reported results:
  • Single model: On the full benchmark in the original comparison, Step-Audio-EditX leads IFR (44.86%) and CR (58.88%), while Ming-UniAudio leads EMR (3.20%); Audio-Omni reaches 4.99% EMR on the separate 801-case, ≤10 s subset. The newer AuK baseline without Prompt Enhancer reports 7.58% EMR on the 1,003-case single-operation subset.
  • Agent / LLM-assisted system: The challenge agent baseline, combining an LLM router with DSP, SAM-Audio, and AuK, reports 7.45% EMR, 44.06% IFR, and 74.63% CR on all 2,000 cases. Separately, AuK-Flash with Prompt Enhancer reaches 13.85% EMR on MMAE-Speech only; these scopes are not directly comparable.

SpeechEditBench

• TL;DR: Separates edit success from linguistic-content preservation in bilingual speech editing through 4,700 cases (≈9.4 h) covering seven atomic attributes and multi-attribute instructions.
• Paper: arXiv; Code: GitHub; Dataset: Hugging Face, v1.1.
• Audio modalities: Speech.
• Editing categories: Acoustic; Semantic; Instance; Composite.
• Evaluation method: Hybrid — ASR, speaker verification, acoustic/prosodic measurements, and a Gemini audio judge produce target success, content-preservation success, and joint success.
• Best reported results:
  • Single model: By task, GPT-Realtime reaches 96.67% content, 68.67% style, and 47.00% paralinguistic joint success; Gemini-Live reaches 27.79% emotion and 11.00% compositional joint success. The newer AuK comparison reports 71.33% prosody joint success, but covers only five task types.

Ming-Freeform-Audio-Edit

• TL;DR: Evaluates timestamp-free, instruction-guided speech changes with ≈3.3k instruction cases across Basic/Full lexical edits and five attribute-control tasks in Chinese and English.
• Paper: arXiv; Code: GitHub; Dataset: Hugging Face; Project Page: Ming-UniAudio.
• Audio modalities: Speech.
• Editing categories: Acoustic; Semantic.
• Evaluation method: Hybrid — Whisper/Paraformer and WavLM measure transcription and speaker preservation; signal measurements assess speed/volume control, and an audio-capable judge assesses emotion/dialect conversion.
• Best reported results:
  • Single model: AuK reports 3.09% / 3.96% WER and 91.47% / 85.25% edit accuracy on Full Chinese / English, averaged over deletion, insertion, and substitution; these lead the checked lexical-editing comparisons.

Step-Audio-Edit-Benchmark

• TL;DR: Evaluates expressive and iterative speech editing using 8 speakers and 8,800 text prompts for emotion, speaking style, and paralinguistics, with reference voices released and output duration dependent on synthesis.
• Paper: arXiv; Code: GitHub; Dataset: Prompt texts · Reference audio; Project Page: Step-Audio-EditX.
• Audio modalities: Speech.
• Editing categories: Semantic.
• Evaluation method: MLLM — Gemini-2.5-Pro measures emotion/style classification accuracy and rates paralinguistic realization on a 1–3 scale.
• Best reported results:
  • Single model: Step-Audio-EditX reports 71.0% emotion accuracy and 66.2% style accuracy after three editing iterations, and 2.89/3 paralinguistic score after one iteration in the published native-input comparison.

LyricEditBench (INTERSPEECH 2026)

• TL;DR: Tests melody-preserving lyric modification with 7,200 bilingual cases, each using a ≤15 s melody reference, across six lyric-editing scenarios and both self-timbre and cross-timbre settings.
• Paper: arXiv; Code: GitHub; Dataset: Hugging Face; Project Page: YingMusic-Singer-Plus.
• Audio modalities: Music.
• Editing categories: Semantic; Instance; Composite (lyric changes together with singer-identity transfer in the cross-timbre setting).
• Evaluation method: Expert models — singing ASR for Phoneme Error Rate (PER), WavLM for speaker similarity, RMVPE for F0 correlation, and VocalVerse2 for vocal quality, supplemented by human listening tests.
• Best reported results:
  • Single model: YingMusic-Singer outperforms Vevo2 on lyric intelligibility, melody adherence, and vocal quality in the published comparison; for Chinese partial substitution, self-timbre, it achieves 2.14% PER and 0.9615 F0 correlation. Vevo2 retains an advantage in speaker similarity on this setting.

ZoME-Bench (ACM MM 2025)

• TL;DR: Provides 1,100 music-editing cases (10 s each; ≈3.1 h counted per case) across instrument, genre, mood, rhythm, melody, and background changes, with captions and instructions supporting both prompt-based and instruction-based evaluation.
• Paper: MEDIC; Code: MEDIC repository (implementation not released) · later evaluation code; Dataset: Hugging Face metadata (source audio retrieved separately from MusicCaps/YouTube); Project Page: MEDIC.
• Audio modalities: Music.
• Editing categories: Semantic; Instance.
• Evaluation method: Expert models — audio–text alignment and perceptual/structural measures such as CLAP, LPAPS, and chroma similarity, supplemented by human ratings.
• Best reported results:
  • Single model / non-agent editor: In the later AnchorSteer instrument-editing comparison, its conditioned variant achieves the highest CLAP (0.395) and GAP (0.279) among the compared methods, while its unconditioned variant preserves more structure (chroma similarity 0.470, versus 0.238 for the conditioned variant).

MelodiaEdit (AAAI 2026)

• TL;DR: Evaluates instrument, genre, and mood changes while preserving musical structure through 2,015 editing pairs drawn from 180 released source clips (≈0.86 h unique audio), combining synthesized and real music.
• Paper: AAAI proceedings; Code: GitHub (data release; evaluation implementation not released); Dataset: Audio and prompts; Project Page: Melodia.
• Audio modalities: Music.
• Editing categories: Semantic; Instance.
• Evaluation method: Expert models — CLAP, LPAPS, chroma similarity, FAD, and combined adherence/preservation scores, supplemented by human listening tests.
• Best reported results:
  • Single model: In the published comparison, Melodia has the highest CLAP (0.39) and lowest LPAPS (3.11) on MelodiaEdit; MusicMagus instead leads chroma similarity (0.73) and FAD (0.57), illustrating the alignment–preservation trade-off.

AvED-Bench (WACV 2026)

• TL;DR: Tests synchronized sound-event and visual-entity replacement using 110 audio-video clips (10 s each; ≈18.3 min) curated from VGGSound with source/target descriptions.
• Paper: arXiv; Code: GitHub; Dataset: Benchmark CSV (source clips retrieved separately from VGGSound/YouTube); Project Page: AvED.
• Audio modalities: Audio (with video input).
• Editing categories: Instance; editing the video alongside the sound does not by itself constitute Composite audio editing.
• Evaluation method: Expert models — embedding-based audio–text/audio–video alignment and perceptual preservation metrics, supplemented by human judgments.
• Best reported results:
  • Single model / non-agent system: The later CoherentAVEdit comparison reports the strongest listening-test results for VACE → CoherentAVEdit: 3.7/5 audio–text fidelity, 3.8/5 audio–visual alignment, and 3.9/5 structure preservation on a six-video subjective subset; this is a sequential, non-agent pipeline, not a single joint model or a full-set aggregate ranking.

AVE-Compass

• TL;DR: Diagnoses instruction following and preservation in free-form audio-video editing through 145 source videos (up to 10 s), 196 instructions, and 2,688 checklist items covering 28 editing operations.
• Paper: arXiv; Code: GitHub; Dataset: Hugging Face; Project Page: AVE-Compass.
• Audio modalities: Speech; Music; Audio (with video input).
• Editing categories: Acoustic; Semantic; Instance; Composite (when the audio request itself crosses categories).
• Evaluation method: Hybrid — checklist-based MLLM judging combines with expert models for audio quality, audio–video/lip synchronization, and visual preservation.
• Best reported results:
  • Single model: On the official leaderboard, Wan2.7 leads overall Editing Intent at 42.4/100 (audio: 24.8); LTX2 has the highest audio-only Editing Intent among the listed single models (26.4/100).
  • Agent: AVE-Agent (Wan) leads with 59.8/100 overall Editing Intent and 50.2/100 audio Editing Intent on the same leaderboard.

📏 Evaluation Metrics

Metrics are grouped by the four evaluation dimensions used in this survey. ↑ / ↓ indicate higher / lower is better. Reference / Inputs lists the information needed alongside the edited output.

Instruction Adherence

Preservation and Locality

For local edits, compare the regions or sources that should remain unchanged.

Temporal and Structural Consistency

Audio Quality and Naturalness

Multi-dimensional Evaluators

Reusable models and toolkits for multi-dimensional assessment of editing results and audio aesthetics.

---

🔮 Challenges and Future Directions

Foundation-model-based audio editing still faces several system-level challenges:

1. Complex editing.
   Real-world audio entangles semantic events, speaker identity, acoustic attributes, background ambience, rhythm, spatial cues, and reverberation. Future systems should support precise source/event localization, attribute-level modification, and reliable preservation of non-target content across speech, music, and general audio.
2. Robustness in open-domain settings.
   Editing models should remain reliable under noise, reverberation, overlapping sources, long-form context, and ambiguous instructions. Better instruction grounding, long-context modeling, iterative refinement, and self-verification are important directions toward robust real-world editing.
3. Faithful and editing-specific evaluation.
   Existing evaluation often mixes generation quality with editing quality. Future benchmarks should explicitly annotate edit targets, operations, preservation regions, and relevant control signals, allowing edit success and non-target preservation to be evaluated separately.
4. Safety, copyright, and misuse prevention.
   Modern editing systems can realistically modify speech content, speaker identity, emotion, environmental sounds, and music. Practical deployment therefore requires complementary mechanisms for provenance, watermarking, manipulated-audio detection, and responsible data licensing.

---

Citation

If You find this survey or repository useful, please cite our paper:

@article{pan2026audio,
  title={Audio Editing in the Era of Foundation Models: A Survey},
  author={Pan, Changhao and Fan, Yifei and Zhuo, Fan and Chen, Yifu and Guo, Wenxiang and Zhang, Yu and Li, Ruiqi and Zhu, Zhiyuan and Yang, Rui and Ji, Shengpeng and others},
  journal={arXiv preprint arXiv:2606.23139},
  year={2026}
}

Contributing

This repo is meant to keep growing. If an audio editing model, dataset, or benchmark is missing, please feel free to open an issue or a pull request.

---

📄 License

Unless otherwise noted below, original content created for this repository is licensed under the MIT License.

The survey paper and content reproduced or adapted from it, including assets/taxonomy_overview.png, assets/train-based.png, and assets/train-free.png, remain under CC BY-NC-SA 4.0. The MIT license does not relicense these materials.

Linked third-party papers, code, models, model weights, datasets, and tools are governed by their respective licenses.