Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skills

Justin Chih-Yao Chen* | Sukwon Yun* | Elias Stengel-Eskin* | Tianlong Chen | Mohit Bansal

*Equal contribution

Table of Contents

• Overview
• Installation
• Running Experiments
  • Shared Arguments
  • Step-by-Step Guide
• Quick Start with Pre-Generated Outputs
• Citation

Overview

This repository contains the implementation of Symbolic Mixture-of-Experts, a novel approach for adaptive skill-based routing to enable scalable heterogeneous reasoning across multiple domains.

Installation

This repository is tested on Python 3.10.12. All dependencies can be installed as follows:

pip install -r requirements.txt

Running Experiments

Shared Arguments

• --task: Specifies which dataset to run. Options include:
  • MMLU_Pro
  • AIME24
  • GPQA
  • MedMCQA
• --gpus: Number of GPUs to use for the experiment
• --seed: Random seed for reproducibility

Step-by-Step Guide

Step 1: Annotate Keywords

Annotate keywords for the validation and testing data:

python annotate_keywords.py

Step 2: Create Model Profiles

Create profiles for all models:

1. Edit run_create_profile.sh to ensure that the --task and --gpus arguments are correct
2. Run the script:

bash run_create_profile.sh

Step 3: Create Aggregator Benchmark

Create the aggregator benchmark for all models:

1. Edit run_bench_aggr.sh to ensure that the --task and --gpus arguments are correct
2. Run the script:

bash run_aggr_bench.sh

Step 4: Recruit Experts

Recruit experts for each instance:

python recruit_agents.py --task GPQA --seed 0

Step 5: Generate Initial Responses

Generate the initial responses from the k experts:

CUDA_VISIBLE_DEVICES=0 python expert_inference.py --task GPQA --gpus 1 --seed 0

Step 6: Aggregate Outputs

Use the aggregator to generate the final output and evaluate the results:

CUDA_VISIBLE_DEVICES=0 python aggregate.py --task GPQA --aggregator QwenR1 --gpus 1 --seed 0

The aggregator we use for each task can be found in the following table (and Table 9 in the paper):

Quick Start with Pre-Generated Outputs

You can skip Steps 1-4 by downloading the stored outputs here (Google Drive).

Extract all files in skill.zip and placing all the folders in the root directory of this project:

unzip skill.zip

After extraction, you can proceed directly to Steps 5-7 to complete the experiment.

Citation

If you find this work useful, please consider citing us:

@article{chen2025skillmoe,
  title={Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skills},
  author={Chen, Justin Chih-Yao and Yun, Sukwon and Stengel-Eskin, Elias and Chen, Tianlong and Bansal, Mohit},
  journal={arXiv preprint arXiv:2503.05641},
  year={2025}
}