Orthrus Reproducibility Study

An independent reproduction study of Orthrus parallel decoding on the published Orthrus-Qwen3-1.7B and Orthrus-Qwen3-8B checkpoints. The study focuses on three questions:

1. Does Orthrus preserve the exact token sequence produced by autoregressive greedy decoding?
2. How much wall-clock acceleration is observed under the reproduced setup?
3. How sensitive are token parity and speed to numerical precision?

The complete results and methodology are available in the technical report. The report is currently written in Russian; the tables and artifact names are language independent.

Main finding: the 8B BF16/FlashAttention-4 run was fast but did not remain token-lossless. Switching to FP32/eager restored 100% token parity with the AR baseline, at the cost of lower throughput.

Results at a glance

Pooled speedup is the sum of AR generation time divided by the sum of Orthrus generation time over the same prompts. Exact token match requires the entire generated token sequence to be identical to AR.

The guarded BF16 extension rechecked low-margin decisions through AR. It nearly restored parity, but 1,436 fallbacks made it slower than the plain AR baseline.

Scope

The 8B reproduction uses greedy decoding (T=0), max_new_tokens=128, and the same 330 prompts for AR and Orthrus. The paper-facing suite contains:

The study also reproduces the MATH-500 block-size ablation for K = 4, 8, 16, 32:

Reproducibility safeguards

The Modal pipeline is designed for paid, interruption-prone GPU runs:

• checkpoint revision and inference-critical files are pinned and hashed;
• the default action only builds a plan and does not allocate an H200;
• a 20-prompt implementation gate precedes bulk runs;
• every prompt/variant is written as an atomic resumable record;
• paid phases are sharded and protected by a workspace lock;
• retries, deadlines, warm-up, timer scope, library versions, and GPU metadata are recorded with the results;
• paper benchmarks and additional domain experiments are aggregated separately.

See EXPERIMENTS.md for the full execution protocol.

Repository layout

.
├── README.md
├── ORTHRUS_REPRODUCIBILITY_REPORT.pdf
├── report/report.md                 # report source
├── main.py                          # inexpensive H200 smoke test
├── experiments.py                   # resumable full Modal pipeline
├── test_experiments.py              # pipeline/unit checks
├── EXPERIMENTS.md                    # runbook and safety limits
└── data/                             # aggregate results for four main runs

Raw model weights and upstream datasets are not redistributed. The repository contains the code, pinned source revisions, aggregate tables, paired timing records, and run summaries needed to audit the reported numbers.

Quick start

Create a local environment and authenticate Modal:

python3 -m venv .venv
./.venv/bin/python -m pip install -r requirements.txt
./.venv/bin/python -m modal setup

Create a Modal secret named huggingface-secret containing a read-only HF_TOKEN. Never commit the token itself.

Run the low-cost smoke test:

./.venv/bin/python -m modal run main.py \
  --prompt-count 1 \
  --max-new-tokens 32

Build the complete experiment plan without allocating an H200:

./.venv/bin/python -m modal run experiments.py \
  --action plan \
  --phase all \
  --max-new-tokens 128

Run local pipeline checks:

./.venv/bin/python -m unittest test_experiments -v

Limitations

This is a best-effort independent reproduction, not an official Orthrus artifact. The paper does not fully specify the inference hardware, precision, attention backend, exact evaluation prompts, output-token limit, timer boundaries, or averaging procedure. Therefore wall-clock values should not be treated as a controlled hardware-for-hardware comparison.

The 1.7B run omits Pseudo2Code because a public equivalent was not available; its performance numbers are indicative rather than directly table-comparable. The strongest reproduced claim is token parity under the documented local setup: 100% in both FP32 runs, but not in the BF16/FA4 run.

References

• Original paper: Orthrus: Towards Evolutionary and Efficient Non-Autoregressive Language Modeling
• Published 8B checkpoint: chiennv/Orthrus-Qwen3-8B

This repository is independent and is not affiliated with the Orthrus authors, Qwen, Hugging Face, or Modal. Upstream models and datasets remain subject to their respective licenses and terms.