LLM-Codec is a neural audio codec checkpoint trained to produce discrete audio tokens that are both reconstructable and easier for autoregressive language models to predict.
Model: https://huggingface.co/voidful/llm-codec
Code: https://github.com/voidful/llm-codec
Usage reference: https://github.com/voidful/Codec-SUPERB
Most neural audio codecs are trained for waveform reconstruction. Spoken language models, however, consume codec tokens with a next-token prediction objective. This mismatch can make acoustically valid variation appear as token uncertainty to the language model.
LLM-Codec adapts a codec with language-model-facing objectives while keeping the deployed codec interface unchanged. The model is trained with:
The deployed codec does not require the auxiliary FTP heads.
This model is intended for research and development in:
It is not a full text-to-speech system by itself. For speech generation, use the codec as the tokenizer/decoder inside a separate speech language modeling pipeline.
Do not use this model for:
The easiest inference path is through the Codec-SUPERB SoundCodec interface.
git clone https://github.com/voidful/Codec-SUPERB.git
cd Codec-SUPERB
pip install -r requirements.txt
export PYTHONPATH=$PWD:$PYTHONPATH
If your environment supports editable installs, this is also convenient:
pip install -e .
Load LLM-Codec through the Codec-SUPERB codec registry:
from SoundCodec import codec
print(codec.list_codec())
model = codec.load_codec("llmcodec")
Encode and reconstruct one audio file:
from SoundCodec import codec
import torchaudio
import soundfile as sf
model = codec.load_codec("llmcodec")
waveform, sample_rate = torchaudio.load("sample_audio.wav")
data_item = {
"audio": {
"array": waveform.numpy()[0],
"sampling_rate": sample_rate,
}
}
units = model.extract_unit(data_item).unit
print("Unit shape:", units.shape)
result = model.synth(data_item, local_save=False)
reconstructed = result["audio"]["array"]
reconstructed_sr = result["audio"].get("sampling_rate", sample_rate)
sf.write("reconstructed.wav", reconstructed, reconstructed_sr)
Codec-SUPERB also provides batch APIs:
from SoundCodec import codec
import torchaudio
model = codec.load_codec("llmcodec")
audio_files = ["audio1.wav", "audio2.wav", "audio3.wav"]
data_list = []
for path in audio_files:
waveform, sample_rate = torchaudio.load(path)
data_list.append({
"id": path,
"audio": {
"array": waveform.numpy()[0],
"sampling_rate": sample_rate,
},
})
batch_units = model.batch_extract_unit(data_list)
batch_audio = model.batch_decode_unit(batch_units)
results = model.batch_synth(data_list, local_save=False)
for item in results:
print(item["unit"].shape, item["audio"]["array"].shape)
For better throughput, group audio samples with similar lengths before batching.
To evaluate LLM-Codec with Codec-SUPERB-tiny:
PYTHONPATH=. python3 scripts/dataset_creator.py \
--dataset voidful/codec-superb-tiny
PYTHONPATH=. python3 scripts/benchmarking.py \
--dataset datasets/voidful/codec-superb-tiny_synth \
--models llmcodec
The model repository provides:
llm-codec.pt<CODEC_*> audio tokensThe codec uses 20,480 audio tokens with the canonical token format:
<CODEC_0>, <CODEC_1>, ..., <CODEC_20479>
The codec was trained on LibriSpeech train-clean-100 with paired transcripts.
The validation split used during training is LibriSpeech validation.
Because training is speech-centric and transcript-supervised, performance may be weaker on non-English speech, conversational speech, music, environmental audio, or audio with strong noise and overlap.
Base components:
Losses:
SALMon speech coherence accuracy after token-level LM training:
| Tokenizer | Overall accuracy |
|---|---|
| WavTok-L | 48.3 |
| BigCodec | 49.4 |
| UniCodec | 50.1 |
| AUV | 49.4 |
| LLM-Codec | 61.6 |
Token-level perplexity on LibriSpeech after 3 epochs of LM training:
| Tokenizer | Eval loss | Perplexity |
|---|---|---|
| WavTok-L | 11.91 | 148,122 |
| UniCodec | 11.92 | 150,197 |
| BigCodec | 11.96 | 156,448 |
| AUV | 11.98 | 159,768 |
| LLM-Codec | 8.44 | 4,617 |
Codec-SUPERB-tiny speech reconstruction:
| Model | Mel lower is better | STFT lower is better | PESQ higher is better | STOI higher is better |
|---|---|---|---|---|
| AUV base | 0.762 | 1.648 | 2.094 | 0.850 |
| LLM-Codec | 0.724 | 1.599 | 2.102 | 0.859 |
transformers artifacts are not a standalone text chatbot;
they accompany the codec/tokenizer workflow.@article{chung2026llm,
title={LLM-Codec: Neural Audio Codec Meets Language Model Objectives},
author={Chung, Ho-Lam and Chen, Yiming and Lee, Hung-yi},
journal={arXiv preprint arXiv:2604.17852},
note = {Model and code available at https://github.com/voidful/llm-codec},
year={2026}
}
If you use the Codec-SUPERB interface or benchmark, please also cite Codec-SUPERB:
@inproceedings{wu-etal-2024-codec,
title = {Codec-SUPERB: An In-Depth Analysis of Sound Codec Models},
author = {Wu, Haibin and Chung, Ho-Lam and Lin, Yi-Cheng and Wu, Yuan-Kuei and Chen, Xuanjun and Pai, Yu-Chi and Wang, Hsiu-Hsuan and Chang, Kai-Wei and Liu, Alexander and Lee, Hung-yi},
booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
year = {2024},
url = {https://aclanthology.org/2024.findings-acl.616},
doi = {10.18653/v1/2024.findings-acl.616},
pages = {10330--10348}
}
37 commits
LLM-Codec is a neural audio codec checkpoint trained to produce discrete audio tokens that are both reconstructable and easier for autoregressive language models to predict.
Model: https://huggingface.co/voidful/llm-codec
Code: https://github.com/voidful/llm-codec
Usage reference: https://github.com/voidful/Codec-SUPERB
Most neural audio codecs are trained for waveform reconstruction. Spoken language models, however, consume codec tokens with a next-token prediction objective. This mismatch can make acoustically valid variation appear as token uncertainty to the language model.
LLM-Codec adapts a codec with language-model-facing objectives while keeping the deployed codec interface unchanged. The model is trained with:
The deployed codec does not require the auxiliary FTP heads.
This model is intended for research and development in:
It is not a full text-to-speech system by itself. For speech generation, use the codec as the tokenizer/decoder inside a separate speech language modeling pipeline.
Do not use this model for:
The easiest inference path is through the Codec-SUPERB SoundCodec interface.
git clone https://github.com/voidful/Codec-SUPERB.git
cd Codec-SUPERB
pip install -r requirements.txt
export PYTHONPATH=$PWD:$PYTHONPATH
If your environment supports editable installs, this is also convenient:
pip install -e .
Load LLM-Codec through the Codec-SUPERB codec registry:
from SoundCodec import codec
print(codec.list_codec())
model = codec.load_codec("llmcodec")
Encode and reconstruct one audio file:
from SoundCodec import codec
import torchaudio
import soundfile as sf
model = codec.load_codec("llmcodec")
waveform, sample_rate = torchaudio.load("sample_audio.wav")
data_item = {
"audio": {
"array": waveform.numpy()[0],
"sampling_rate": sample_rate,
}
}
units = model.extract_unit(data_item).unit
print("Unit shape:", units.shape)
result = model.synth(data_item, local_save=False)
reconstructed = result["audio"]["array"]
reconstructed_sr = result["audio"].get("sampling_rate", sample_rate)
sf.write("reconstructed.wav", reconstructed, reconstructed_sr)
Codec-SUPERB also provides batch APIs:
from SoundCodec import codec
import torchaudio
model = codec.load_codec("llmcodec")
audio_files = ["audio1.wav", "audio2.wav", "audio3.wav"]
data_list = []
for path in audio_files:
waveform, sample_rate = torchaudio.load(path)
data_list.append({
"id": path,
"audio": {
"array": waveform.numpy()[0],
"sampling_rate": sample_rate,
},
})
batch_units = model.batch_extract_unit(data_list)
batch_audio = model.batch_decode_unit(batch_units)
results = model.batch_synth(data_list, local_save=False)
for item in results:
print(item["unit"].shape, item["audio"]["array"].shape)
For better throughput, group audio samples with similar lengths before batching.
To evaluate LLM-Codec with Codec-SUPERB-tiny:
PYTHONPATH=. python3 scripts/dataset_creator.py \
--dataset voidful/codec-superb-tiny
PYTHONPATH=. python3 scripts/benchmarking.py \
--dataset datasets/voidful/codec-superb-tiny_synth \
--models llmcodec
The model repository provides:
llm-codec.pt<CODEC_*> audio tokensThe codec uses 20,480 audio tokens with the canonical token format:
<CODEC_0>, <CODEC_1>, ..., <CODEC_20479>
The codec was trained on LibriSpeech train-clean-100 with paired transcripts.
The validation split used during training is LibriSpeech validation.
Because training is speech-centric and transcript-supervised, performance may be weaker on non-English speech, conversational speech, music, environmental audio, or audio with strong noise and overlap.
Base components:
Losses:
SALMon speech coherence accuracy after token-level LM training:
| Tokenizer | Overall accuracy |
|---|---|
| WavTok-L | 48.3 |
| BigCodec | 49.4 |
| UniCodec | 50.1 |
| AUV | 49.4 |
| LLM-Codec | 61.6 |
Token-level perplexity on LibriSpeech after 3 epochs of LM training:
| Tokenizer | Eval loss | Perplexity |
|---|---|---|
| WavTok-L | 11.91 | 148,122 |
| UniCodec | 11.92 | 150,197 |
| BigCodec | 11.96 | 156,448 |
| AUV | 11.98 | 159,768 |
| LLM-Codec | 8.44 | 4,617 |
Codec-SUPERB-tiny speech reconstruction:
| Model | Mel lower is better | STFT lower is better | PESQ higher is better | STOI higher is better |
|---|---|---|---|---|
| AUV base | 0.762 | 1.648 | 2.094 | 0.850 |
| LLM-Codec | 0.724 | 1.599 | 2.102 | 0.859 |
transformers artifacts are not a standalone text chatbot;
they accompany the codec/tokenizer workflow.@article{chung2026llm,
title={LLM-Codec: Neural Audio Codec Meets Language Model Objectives},
author={Chung, Ho-Lam and Chen, Yiming and Lee, Hung-yi},
journal={arXiv preprint arXiv:2604.17852},
note = {Model and code available at https://github.com/voidful/llm-codec},
year={2026}
}
If you use the Codec-SUPERB interface or benchmark, please also cite Codec-SUPERB:
@inproceedings{wu-etal-2024-codec,
title = {Codec-SUPERB: An In-Depth Analysis of Sound Codec Models},
author = {Wu, Haibin and Chung, Ho-Lam and Lin, Yi-Cheng and Wu, Yuan-Kuei and Chen, Xuanjun and Pai, Yu-Chi and Wang, Hsiu-Hsuan and Chang, Kai-Wei and Liu, Alexander and Lee, Hung-yi},
booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
year = {2024},
url = {https://aclanthology.org/2024.findings-acl.616},
doi = {10.18653/v1/2024.findings-acl.616},
pages = {10330--10348}
}
37 commits