Make Router Evaluation Open and Standardized

RouterArena is an open evaluation platform and leaderboard for LLM routers—systems that automatically select the best model for a given query. As the LLM ecosystem diversifies with models varying in size, capability, and cost, routing has become critical for balancing performance and cost. Yet, LLM routers currently lack a standardized evaluation framework to assess how effectively they trade off accuracy, cost, and other related metrics.

RouterArena bridges this gap by providing an open evaluation platform and benchmarking framework for both open-source and commercial routers. It has the following key features:

• 🌍 Diverse Data Coverage: A principly-constructed, diverse evaluation dataset spanning 9 domains and 44 categories with easy, medium, and hard difficulty levels.
• 📊 Comprehensive Metrics: Five router-critical metrics measuring accuracy, cost, optimality, robustness, and latency.
• ⚙️ Automated Evaluation: An automated evaluation framework to simplify the evaluation process for open-source and commercial routers.
• 🏆 Live Leaderboard: A live leaderboard to track the performance of routers across multiple dimensions.

We aim for RouterArena to serve as a foundation for the community to evaluate, understand, and advance LLM routing systems.

[!IMPORTANT] RouterArena is an evaluation-only dataset. Submissions that train, fit, or tune any router component on RouterArena data (including the label files) will be rejected, and any accepted submission found in violation will be withdrawn.

Current Leaderboard

For more details, please see our website and blog.

Under review: Paix2 (#164) is temporarily removed from the ranking pending an integrity review. See #211.

🎓 Open-source  💼 Closed-source 

Evaluating Your Router

1. Setup

Step 1.1: Install uv and RouterArena

curl -LsSf https://astral.sh/uv/install.sh | sh
cd RouterArena
uv sync

Step 1.2: Download Dataset

Download the dataset from HF dataset.

uv run python ./scripts/process_datasets/prep_datasets.py

Step 1.3: Set Up API Keys (Optional)

In the project root, copy .env.example as .env and update the API keys in .env. This step is required only if you use our pipeline for LLM inferences.

# Example .env file
OPENAI_API_KEY=<Your-Key>
ANTHROPIC_API_KEY=<Your-Key>
# ...

See the ModelInference class for the complete list of supported providers and required environment variables. You can extend that class to support more models, or submit a GitHub issue to request support for new providers.

2. Get Routing Decisions

Follow the steps below to obtain your router's model choices for each query. Start with the sub_10 split (a 10% subset) for local testing. Once your setup works, you can evaluate:

• on the full dataset for full local evaluation and official leaderboard submission.
• on the robustness dataset for robustness evaluation.

Step 2.1: Prepare Config File

Create a config file in ./router_inference/config/<router_name>.json. An example config file is included here.

{
  "pipeline_params": {
      "router_name": "your-router",
      "router_cls_name": "your_router_class_name",
      "models": [
          "gpt-4o-mini",
          "claude-3-haiku-20240307",
          "gemini-2.0-flash-001"
      ]
  }
}

For each model in your config, add an entry with the pricing per million tokens in this format at model_cost/model_cost.json:

{
  "gpt-4o-mini": {
    "input_token_price_per_million": 0.15,
    "output_token_price_per_million": 0.6
  },
}

[!NOTE] Ensure all models in your above config files are listed in ./universal_model_names.py. If you add a new model, you must also add the API inference endpoint in llm_inference/model_inference.py.

Step 2.2: Create Your Router Class and Generate Prediction File

Create your own router class by inheriting from BaseRouter and implementing the _get_prediction() method. See router_inference/router/example_router.py for a complete example.

Then, modify router_inference/router/__init__.py to include your router class:

# Import your router class
from router_inference.router.my_router import MyRouter

__all__ = ["BaseRouter", "ExampleRouter", "MyRouter"]

Finally, generate the prediction file:

uv run python ./router_inference/generate_prediction_file.py your-router [sub_10|full|robustness]

[!NOTE]

• The <your-router> argument must match your config filename (without the .json extension). For example, if your config file is router_inference/config/my-router.json, use my-router as the argument.
• Your _get_prediction() method must return a model name that exists in your config file's models list. The base class will automatically validate this.

Step 2.3: Validate Config and Prediction Files

uv run python ./router_inference/check_config_prediction_files.py your-router [sub_10|full|robustness]

This script checks: (1) all model names are valid, (2) prediction file has correct size (809 for sub_10, 8400 for full, 420 for robustness), and (3) all entries have valid global_index, prompt, and prediction fields.

3. Run LLM Inference

Run the inference script to make API calls for each query using the selected models:

uv run python ./llm_inference/run.py your-router

The script loads your prediction file, makes API calls using the models specified in the prediction field, and saves results incrementally. It uses cached results when available and saves progress after each query, so you can safely interrupt and resume. Results are saved to ./cached_results/ for reuse across routers.

[!NOTE]

• For robustness evaluation, we only measure the model-selection flip ratio after adding noise to the original prompt, so no additional LLM inference is required for this stage.

4. Run Router Evaluation

As the last step, run the evaluation script:

uv run python ./llm_evaluation/run.py your-router [sub_10|full|robustness]

[!TIP]

• Use sub_10 or full to evaluate on those datasets.
• Use robustness to run robustness-only evaluation (expects <router_name>-robustness.json).

Submitting to the leaderboard

To get your router on the leaderboard, you can open a Pull Request with your router's prediction file to trigger our automated evaluation workflow. Details are as follows:

1. Add your files:
   • router_inference/config/<router_name>.json - Your router configuration
   • router_inference/predictions/<router_name>.json - Your prediction file with generated_result fields populated
   • router_inference/predictions/<router_name>-robustness.json - Your prediction file for robustness evaluation, no generated_result fields needed
2. Open a Pull Request to main branch and call /evaluate in the PR comment
   • When the PR is ready for evaluation, call /evaluate in the PR comment to trigger the evaluation workflow. See an example here.
   • The automated workflow will:
     • Validate your submission
     • Run evaluation on the full dataset
     • Post results as a comment on your PR
     • Update the leaderboard upon approval

The Figure below shows the evaluation pipeline.

Contributing

We welcome and appreciate contributions and collaborations of any kind.

We use pre-commit to ensure a consistent coding style. You can set it up by

pip install pre-commit
pre-commit install

Before pushing your code, run the following and make sure your code passes all checks.

pre-commit run --all-files

Contacts

Feel free to contact us for contributions and collaborations.

Yifan Lu (yifan.lu@rice.edu)
Rixin Liu (rixin.liu@rice.edu)
Jiarong Xing (jxing@rice.edu)

Citation:

If you find our project helpful, please give us a star and cite us by:

@misc{lu2025routerarenaopenplatformcomprehensive,
  title        = {RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers},
  author       = {Yifan Lu and Rixin Liu and Jiayi Yuan and Xingqi Cui and Shenrun Zhang and Hongyi Liu and Jiarong Xing},
  year         = {2025},
  eprint       = {2510.00202},
  archivePrefix= {arXiv},
  primaryClass = {cs.LG},
  url          = {https://arxiv.org/abs/2510.00202}
}