Orthrus Experiments

Reproducible Kaggle experiments for evaluating Orthrus, a dual-view model that combines autoregressive (AR) decoding with parallel diffusion proposals and batched AR verification.

The experiments focus on three questions:

1. Does Orthrus reproduce the exact token IDs generated by the original Qwen3 AR model?
2. How do precision, attention backend, and diffusion block size affect throughput and algorithmic efficiency?
3. Can the block size be adapted dynamically from the observed draft acceptance rate?

The tested checkpoints are:

• AR baseline: Qwen/Qwen3-1.7B
• Orthrus: chiennv/Orthrus-Qwen3-1.7B

The experiments use examples from:

• GSM8K
• MATH-500
• HumanEval

Repository contents

Environment

The scripts are designed for a Kaggle notebook with GPU and Internet access enabled.

The experiments reported below were run with:

GPU: Tesla T4, compute capability 7.5
torch: 2.10.0+cu128
transformers: 5.8.1

Install the required Python packages:

pip install -U "transformers==5.8.1" "datasets>=4.0.0" "accelerate>=1.0.0"

The Orthrus checkpoint uses custom Hugging Face model code and is loaded with trust_remote_code=True. Review and pin the model revision when running in a security-sensitive environment.

An authenticated Hugging Face token is optional but avoids anonymous Hub rate limits:

import os
os.environ["HF_TOKEN"] = "..."

Quick start on Kaggle

git clone https://github.com/NikitaKukuzey/orthrus_exps.git
cd orthrus_exps
pip install -U "transformers==5.8.1" "datasets>=4.0.0" "accelerate>=1.0.0"

Run a small smoke test before a full benchmark:

python orthrus_lossless_kaggle_v2.py \
  --samples-per-dataset 1 \
  --max-new-tokens 64 \
  --repeats 1 \
  --dtype float16 \
  --attn-implementation sdpa

1. Token-level losslessness and speed

orthrus_lossless_kaggle_v2.py evaluates three modes:

1. Original Qwen3 autoregressive decoding.
2. The Orthrus checkpoint with use_diffusion_mode=False.
3. The same Orthrus checkpoint with use_diffusion_mode=True.

Generation is greedy (temperature = 0) so exact token-ID comparison is meaningful. Models are loaded sequentially to fit on a Kaggle GPU, and CUDA is synchronized around every timed generation call.

Recommended T4 run:

python orthrus_lossless_kaggle_v2.py \
  --samples-per-dataset 1 \
  --max-new-tokens 256 \
  --repeats 2 \
  --dtype float16 \
  --attn-implementation sdpa

Slow FP32 numerical control:

python orthrus_lossless_kaggle_v2.py \
  --samples-per-dataset 1 \
  --max-new-tokens 256 \
  --repeats 2 \
  --dtype float32 \
  --attn-implementation eager

Illustrative T4 results

These results use only the first example from each dataset and should be treated as diagnostic observations, not universal performance claims.

On a Tesla T4, FP16 is the appropriate performance dtype. T4 has native FP16 Tensor Cores but no native BF16 Tensor Cores. BF16 execution on this GPU is useful as a numerical diagnostic, not as a performance configuration.

The benchmark writes:

orthrus_lossless_results.json
orthrus_lossless_summary.csv

2. First-mismatch logit diagnostics

orthrus_first_mismatch_logits.py isolates the first token where sequential Orthrus AR and Orthrus diffusion generation diverge. It ensures that both paths have the same prompt and generated prefix, then captures the vocabulary-logit rows that predict the mismatching token.

python orthrus_first_mismatch_logits.py \
  --dtype bfloat16 \
  --attn-implementation eager \
  --max-new-tokens 256

The diagnostic reports:

• actual token selected by each GPU generation path;
• top-candidate margins and exact BF16 ties;
• logits of the two competing tokens;
• maximum absolute difference between the logit vectors;
• sequential query length and verifier block length.

Observed BF16 mismatches on T4

All three divergences occurred under an identical generated prefix. Sequential AR used query length = 1, while the batched verifier used query length = 32.

At logit magnitudes between 32 and 64, adjacent BF16 values are spaced by 0.25. The observed differences can therefore create or remove exact ties and change greedy argmax, even when the two paths are mathematically equivalent in exact arithmetic.

Output:

orthrus_first_mismatch_logits.json

3. Fixed block-size sweep

orthrus_block_size_kaggle.py evaluates K = 4, 8, 16, 32 in FP16 on 30 prompts: ten from each dataset.

python orthrus_block_size_kaggle.py \
  --block-sizes 4 8 16 32 \
  --samples-per-dataset 10 \
  --max-new-tokens 256 \
  --repeats 2 \
  --attn-implementation sdpa

Timed generation and acceptance tracing are separate. The additional diagnostic trace is not included in tokens/second.

T4 block-size results

Increasing K from 16 to 32 improved pooled throughput by approximately 3.7%, while doubling the maximum proposed block length. The identical p90 value of 15 indicates that most K=32 cycles did not benefit from the additional proposal capacity.

Exact vs K=4 compares complete generated token sequences with the K=4 result. It is not a comparison with Qwen AR and is not an accuracy metric. K=4 reports 30/30 because it is the reference.

Outputs:

orthrus_block_size_sweep.json
orthrus_block_size_sweep_details.csv
orthrus_block_size_sweep_summary.csv

4. Dynamic block size

orthrus_dynamic_block_kaggle.py adapts K after every verifier cycle. Let A be the number of accepted draft tokens and P the number proposed in the current cycle:

r = A / P

r >= 0.75  -> next K = min(32, current K * 2)
r <= 0.25  -> next K = max(4, current K // 2)
otherwise  -> keep the current K

Run the default policy in FP16:

python orthrus_dynamic_block_kaggle.py \
  --samples-per-dataset 10 \
  --max-new-tokens 256 \
  --repeats 2 \
  --initial-k 16 \
  --attn-implementation sdpa

The script reports overall and per-dataset:

• tokens/second;
• effective and inclusive TPF;
• accepted draft tokens per cycle;
• overall draft acceptance rate;
• arithmetic mean K per cycle;
• progress-weighted mean K;
• cycle counts at K=4, 8, 16, and 32;
• increase, decrease, and hold transition counts.

Outputs:

orthrus_dynamic_block_results.json
orthrus_dynamic_block_details.csv
orthrus_dynamic_block_summary.csv

Metric definitions

Accepted draft tokens per cycle

The number of consecutive diffusion-proposed tokens that match the batched AR verifier before the first mismatch. The verifier correction token is not included.

Generated progress per cycle

The number of new output tokens committed by a cycle. It normally equals:

accepted draft tokens + 1 verifier correction token

It can be smaller in the final cycle because of EOS or the max_new_tokens boundary.

Effective TPF

Steady-state Tokens Per Forward Pass:

effective_tpf = (generated_tokens - 1) / (2 * generation_cycles)

The initial prompt prefill and first generated token are excluded. Every Orthrus generation cycle performs one diffusion pass and one verifier pass.

Inclusive TPF

TPF including the initial prefill:

inclusive_tpf = generated_tokens / (1 + 2 * generation_cycles)

Tokens per second

Generated tokens divided by CUDA-synchronized wall time. Dataset/model loading, tokenization, warm-up, result serialization, and separate diagnostic traces are excluded unless a script states otherwise.

p50 and p90

The 50th and 90th percentiles of accepted draft tokens across all generation cycles in the selected sample.

Mean K

The arithmetic mean of the block sizes used across dynamic generation cycles. progress_weighted_mean_k additionally weights each K by the number of output tokens committed during that cycle.

Reproducibility notes and limitations

• Results are specific to the GPU, precision, attention backend, software versions, prompt formatting, and checkpoint revision.
• FP16, BF16, and FP32 runs should not be mixed in one aggregate comparison.
• Mathematically equivalent sequential and batched operations are not guaranteed to be bitwise identical in finite precision.
• A benchmark-level accuracy match does not imply token-level sequence identity.
• Block-size output equality is reported separately because changing K changes the verifier tensor shape and can change low-precision logits.
• The scripts use greedy decoding. Independent stochastic generations are not expected to produce identical samples even if they target the same distribution.
• These experiments measure inference behavior; they do not retrain the Orthrus diffusion branch for smaller or dynamic K.
• The reported T4 tables are preliminary measurements from this repository's experimental setup, not official Orthrus benchmark results.

References

• Orthrus paper
• Official Orthrus repository
• Orthrus-Qwen3-1.7B checkpoint
• Qwen3-1.7B checkpoint