audio2chart

audio2chart is an open-source deep learning framework for audio to chart generation, converting raw audio into structured .chart files used in Guitar Hero style rhythm games.
A complete description of the methodology, architecture, and experiments can be found in our arXiv publication.

The repository provides a full codebase for both training and inference, including data processing pipelines, neural network architectures, and ready to use scripts for generating playable charts from real songs.

Overview

The repository contains everything needed to:

• Run pretrained models to generate .chart files from audio
• Train or reproduce the baseline sequence model

The main use case is simple:

Input: an .wav, or .mp3 audio file
Output: a playable .chart file compatible with Clone Hero

---

Quick Start on Google Colab

You can try audio2chart instantly in your browser without setup.

1. Click on the following botton:

   

2. The notebook will:

   • Install dependencies
   • Download a pretrained model from Hugging Face
   • Transcribe your own .mp3 or .wav file into a .chart file

---

Local Installation

To run locally (with GPU support):

git clone https://github.com/3podi/audio2chart.git
cd audio2chart
pip install -r requirements.txt

---

Inference

generate.py allows you to transcribe an audio file into a .chart file using a pretrained model from Hugging Face.

python generate.py path/to/audio.mp3

This will automatically download the default model (3podi/charter-v1.0-40-M-best-acc-nonpad), generate chart tokens from the input audio, and save the resulting .chart file in the current directory.

Full example with custom parameters:

python generate.py path/to/audio.mp3 \
  --model_name 3podi/charter-v1.0-40-M-best-acc-nonpad \
  --temperature <float_temperature> \
  --top_k <int_topk> \
  --name "<song_title>" \
  --artist "<artist_name>" \
  --album "<album_name>" \
  --genre "<genre_name>" \
  --charter "<charter_name>" \
  --output <output_path>.chart

The script will:

1. Load the specified pretrained model
2. Encode the input audio
3. Generate dense token sequences conditioned on audio
4. Decode them into time-aligned notes
5. Write the final .chart file to the output path

---

Baseline Model

baseline.py implements a simple Transformer decoder that predicts chart tokens without any audio conditioning.

The baseline can be trained and evaluated on publicly available tokenized chart datasets and serves as a reference model for pure symbolic note prediction. The dataset is available at:
👉 3podi/audio2chart-charts

It contains a charts.zip archive with preprocessed chart data ready for training.

To download and extract the dataset:

huggingface-cli download 3podi/audio2chart-charts charts.zip --repo-type dataset -d <data_path>
unzip <data_path>/charts.zip -d <data_path>/charts

Running the Baseline

python baseline.py

Training and evaluation are configured via Hydra, which automatically loads the default configuration files located in the configs/ directory.
All parameters such as learning rate, batch size, epochs, or save directory, are defined in the YAML files.

To override a parameter at runtime, simply append it to the command line, for example:

python baseline.py root_folder=<data_path> is_discrete=True window_seconds=30 grid_ms=40

---

Audio-Conditioned Model

main.py is used to train or evaluate the audio-conditioned Transformer model that maps encoded audio features to chart tokens.

The default uses frozen pretrained Encodec features. A trainable SEANet encoder can be selected through Hydra:

python main.py model=audio_discrete

Both encoder choices use the same discrete charting Lightning module. The chart-only autoregressive baseline remains available through baseline.py.

The pretrained weights are provided on Hugging Face (see below), and inference is fully supported through generate.py.

---

🤗 Hugging Face Models

You can load and run the pretrained Charter model directly from Hugging Face.

from inference.engine import Charter

# Load the pretrained audio-to-chart model
model = Charter.from_pretrained("3podi/charter-v1.0-40-M-best-acc-nonpad")

# Generate chart tokens from an audio file
seqs = model.generate("path/to/song.mp3")

The default model 3podi/charter-v1.0-40-M-best-acc-nonpad offers a strong balance between quality and speed.

Other available models vary by:

• Time resolution: 20 ms or 40 ms (controls temporal precision)
• Model size: S (~25 M params) or M (~225 M params)
• Checkpoint type: best-acc / best-acc-nonpad

You can explore all model variants in the
👉 Charter v1.0 collection on Hugging Face.

---

Repository Structure

audio2chart/
├── inference/
│   ├── model_inference.py     # Inference model with KV-cache support
│   ├── layers.py              # Self-contained inference model primitives
│   └── engine.py              # Inference engine
├── chart/
│   ├── tokenizer.py           # Tokenization utilities
│   ├── time_conversion.py     # Convert note times to tick values and viceversa
│   └── chart_writer.py        # Writes decoded charts to .chart format
├── dataloader/
│   ├── convert_to_raw.py      # Script to convert audio to raw format
│   ├── audio_loader.py        # Dataloader for audio conditioned training
│   ├── notes_loader.py        # Dataloader for notes only training
│   └── utils_dataloader.py    # Dataset utils
├── modules/
│   ├── models.py              # torch.nn models
│   ├── trainer.py             # Lightning training modules
│   ├── training_transformer.py # Audio-conditioned training model
│   ├── utils_train.py         # Training utils
│   └── run_utils.py           # Training run setup and lifecycle
├── notebooks/
│   └── generating.py          # Colab notebook for charting
├── baseline.py                # Baseline training script
├── main.py                    # Audio-conditioned training script
├── generate.py                # Main inference entry point
└── requirements.txt

---

Citation

If you use audio2chart or its models in your work, please consider citing:

@misc{tripodi2025audio2chartendendaudio,
      title={audio2chart: End to End Audio Transcription into playable Guitar Hero charts}, 
      author={Riccardo Tripodi},
      year={2025},
      eprint={2511.03337},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2511.03337}, 
}

---

Contact

For questions, discussions, or contributions, open an Issue on GitHub.

---

Acknowledgments

This project was supported by AMD and the AMD Developer Cloud, whose compute resources made training and experimentation possible.

---

Tests and benchmarks

Run the dependency-light chart, tokenizer, and timing suite with:

python -m unittest tests.test_chart_pipeline

After installing the pinned environment (pip install -r requirements.txt), run the full suite with:

python -m unittest discover -s tests -p 'test_*.py'

Dependency-heavy tests skip with a clear reason if the training stack is unavailable. Performance comparisons are kept separate from unit tests:

python tests/benchmark_chart_pipeline.py
python tests/benchmark_chart_pipeline.py --json
python tests/benchmark_chart_pipeline.py --max-ratio 1.25

The benchmark checks reference/refactored output equivalence before reporting median timing ratios for small, typical, and large synthetic charts. The optional --max-ratio makes it fail when any ratio exceeds the supplied threshold.

---

⭐ If you find this project useful, please give it a star!