xRouteBench — LLM Routing Benchmark

xRouteBench is a benchmark for training and evaluating LLM routers — systems that pick the best LLM from a candidate pool for each incoming query, trading off performance vs. price cost.

Every query in each scenario was executed against all 18 candidate LLMs, recording each model's response, task performance, token usage, and latency. A router learns from the train split which model to pick, and is evaluated on test.

Scenarios (configs)

Schema (routing data)

Each row = one (query, candidate model) pair:

Raw query configs (*_queries)

Each scenario also ships its raw query set (no model executions) as a <scenario>_queries config with train / valid / test splits — use these to run your own candidate models from scratch. Fields: task_name, query, ground_truth, metric, choices, task_id (+ conversation_id/category for the memory scenarios). The memory queries are the RAG top-k=5 turn-pair variant used in the published experiments. Note: the valid split exists only here; the routing-data configs have train/test.

Candidate pool & pricing (llm_candidates config)

18 models spanning $0.05–$1.25 per 1M input tokens (25× spread) served via Together AI / NVIDIA NIM. Row cost = input_tokens × input_price/1e6 + output_tokens × output_price/1e6.

Usage

from datasets import load_dataset

ds = load_dataset("ulab-ai/xRouteBench", "llmrouter_generic")   # any config name above
train, test = ds["train"], ds["test"]

pricing = load_dataset("ulab-ai/xRouteBench", "llm_candidates")["train"]

Composite reward for cost-aware routing (GraphRouter-style):

reward = α · norm(performance) − β · norm(price_cost)

💻 Code

The official implementation of LLMRouter, including router implementations, training and evaluation pipelines, and utilities for working with xRouteBench, is available on GitHub:

👉 ulab-uiuc/LLMRouter

Notes

• Query embeddings (.pt) are not included; they can be regenerated from the query field with any sentence encoder.
• The memory scenarios use the RAG top-k=5 turn-pair context variant.

📚 Citation

If you find xRouteBench useful for your research or projects, please cite it as:

@article{feng2026llmrouter,
  title={LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers},
  author={Feng, Tao and Yu, Fangxu and Zhang, Haozhen and Dai, Zhongjie and Yuan, Liangqi and Lei, Zijie and Zhang, Weizhi and Zhu, Kunlun and Yue, Haodong and Xuan, Keyang and others},
  journal={arXiv preprint arXiv:2608.06867},
  year={2026}
}