PAST: Phonetic-Acoustic Speech Tokenizer

Authors: Nadav Har-Tuv, Or Tal, Yossi Adi
Affiliation: The Hebrew University of Jerusalem

📄 Paper PDF | 🌐 Project Page | 💻 Code

Abstract

We present PAST, a novel end-to-end framework that jointly models phonetic information alongside signal reconstruction, eliminating the need for external pretrained models. Unlike previous approaches that rely on pretrained self-supervised models, PAST employs supervised phonetic data, directly integrating domain knowledge into the tokenization process via auxiliary tasks. Additionally, we introduce a streamable, causal variant of PAST, enabling real-time speech applications. Results demonstrate that PAST surpasses existing evaluated baseline tokenizers across common evaluation metrics, including phonetic representation and speech reconstruction. Notably, PAST also achieves superior performance when serving as a speech representation for speech language models, further highlighting its effectiveness as a foundation for spoken language generation.

Samples

Audio samples are available on our project demo page.

Model List

---

Usage

Pre-requisites

Install

conda create -n past_env python=3.10 -y
conda activate past_env
pip install git+https://github.com/slp-rl/PAST.git

Clone

git clone https://github.com/slp-rl/PAST.git
conda create -n past_env python=3.10 -y
conda activate past_env
pip install -r requirements.txt

Inference

# ---------------
# load PAST model
# ---------------

from past.models.past_model import PastModel
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
model = PastModel.from_pretrained("PAST", device=device)  # one of ['PAST', 'PAST_streamable']


# ----------------------------------------------------------------------
# Run on audio: PAST expects a batched input format [Batch, Channels, T]
# ----------------------------------------------------------------------
import torchaudio

def read_one_wav(path, target_sr):
    wav, sr = torchaudio.load(path)
    if sr != target_sr:
        wav = torchaudio.transforms.Resample(sr, target_sr)(wav)
    if wav.shape[0] == 2:
        wav = wav[:1]
    return wav.unsqueeze(0)

wav = read_one_wav("path/to/audio.wav", model.sample_rate).to(device)

with torch.no_grad():
    codes, scale = model.encode(wav)
    reconstructed = model.decode(codes, scale)

Evaluation

See Eval README

---

Results (from the paper)

Phonetic Information

Reconstruction Quality

Speech Language Modeling (sWUGGY)

---

Citation

If you use PAST in your work, please cite:

@article{har2025past,
    title={Past: Phonetic-acoustic speech tokenizer},
    author={Har-Tuv, Nadav and Tal, Or and Adi, Yossi},
    journal={arXiv preprint arXiv:2505.14470},
    year={2025}
  }