---

📋 Table of Contents

• Installation
• Core Code
• Tests
• Supported LLMs
• Inference
  • Evaluation
  • Speed Analysis
  • Free Chat
• Acknowledgement

---

🛠️ Installation

conda create -n racer python=3.10
conda activate racer
cd RACER
pip install -r requirements.txt

---

🧩 Core Code

The speculative-decoding automaton is the algorithmic core of RACER:

racer/automaton/automaton.cpp is generated by Cython and is not source of truth.

---

🧪 Tests

Rebuild the automaton extension, then run the CPU-only paper-alignment unit tests:

pip install -e ./racer/automaton
python -m unittest tests.test_automaton_paper_alignment -v

These tests cover Logits Tree breadth (Eq. 3 / Algorithm 1), Tree Attention (Eq. 2), default border depth, constructor clamping, and draft-capacity bounds. They do not require a GPU.

---

🤖 Supported LLMs

---

🚀 Inference

Evaluation

Run benchmarks to measure mean accepted tokens and speedup ratios:

# Example: Evaluate Qwen3-1.7B on MGSM-ZH
CUDA_VISIBLE_DEVICES=0 python -m evaluation.inference_racer \
  --model-path qwen/qwen3-1.7b \
  --model-id qwen3-1.7b-racer \
  --bench-name mgsm

--bench-name accepts: spec_bench | human_eval | mgsm | gsm8k | math | aime |  (default: spec_bench)

---

Speed Analysis

Measure decoding throughput and compare against baselines:

# Single-file speed analysis
python evaluation/speed.py <file_path>

# Compare with a baseline (calculates speedup ratio)
python evaluation/speed.py <file_path> --base <base_path>

# Batch analysis across an entire folder
python evaluation/batch_speed.py <folder> --base <base_path>

---

Free Chat

Accepted draft tokens (excluding the sampled next token) are highlighted in green.

Terminal CLI

CUDA_VISIBLE_DEVICES=0 python -m racer.inference.cli --model-path qwen/qwen3-1.7b

# Optional arguments:
#   --temperature   Sampling temperature  (default: 0.7)
#   --top-p         Nucleus sampling p    (default: 0.8)

Web UI

CUDA_VISIBLE_DEVICES=0 python -m racer.inference.webui --model-path qwen/qwen3-1.7b

---

Acknowledgement

We would like to thank openPangu for its support. openPangu is a trademark of Huawei Technologies Co., Ltd. For more information, please refer to the official openPangu repository and the accompanying license files.