[!TIP] Support this work: donate.sybilsolutions.ai

REAP surfaces: GLM | MiniMax | Qwen | Gemma | Paper | Code | PR17 | Cerebras Collection

REAP Calibration Dataset v1

Benchmark-free calibration dataset for REAP (Routing-Enhanced Activation Pruning) of Mixture-of-Experts language models.

What This Dataset Does

REAP prunes MoE models by removing experts that rarely activate. To decide which experts are safe to remove, REAP needs to observe which experts fire on diverse inputs. This dataset provides those inputs.

This is NOT training data. No model weights are updated. The dataset is fed through the model in inference mode to record expert routing statistics. Those statistics then guide the pruning decisions.

Key Property: Zero Benchmark Contamination

This dataset was specifically curated to exclude all common evaluation benchmarks, ensuring REAP pruning decisions are not biased toward benchmark-specific patterns.

Excluded Benchmarks

HumanEval, MBPP, EvalPlus, SWE-bench (all variants), TerminalBench, GSM8K, MATH-500, GAIA, KernelBench, ARC, BoolQ, HellaSwag, WinoGrande, MMLU (except physics/chemistry for science coverage), TruthfulQA, PIQA, OpenBookQA, MathQA, LiveCodeBench.

Dataset Composition

23,088 samples across 10 domains, weighted toward coding and tool-use workloads:

Domain Breakdown

Source Datasets

REAP Packing Strategy

Per the REAP paper, for models ≥110B parameters:

• No packing — each sample is its own sequence
• Max sequence length: 16,384 tokens
• Samples longer than 16K tokens are truncated
• The REAP observer handles tokenization and batching at runtime
• Batch size: 8 sequences per forward pass

For models <110B parameters, the paper recommends packing multiple samples to fill 2,048-token sequences.

Format

JSONL with fields:

{
  "id": "function_calling_0",
  "domain": "function_calling",
  "repo_id": "Salesforce/xlam-function-calling-60k",
  "subset": "default",
  "text": "..."
}

Usage with REAP

python scripts/run_qwen35_layerwise_observations_pr17.py \
  --dataset-jsonl calibration-v1.jsonl \
  --max-tokens 16384 \
  --batch-size 8 \
  --observation-sequence-chunk-size 1 \
  --max-group-batches 20 \
  --checkpoint-every-samples 800

Models Calibrated With This Dataset

• Qwen3.5-122B-A10B-REAP-20 (20% pruned, 97.9% capability retained)
• Qwen3.5-122B-A10B-REAP-30 (30% pruned)
• Qwen3.5-122B-A10B-REAP-40 (40% pruned)
• Qwen3.5-397B-A17B (observations in progress)

Maintainer

• Author: 0xSero
• Organization: Sybil Solutions
• Project: REAP PR17

Citation

@article{lu2025reap,
  title={Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture of Experts},
  author={Lu, Xudong and Qiu, Liu and Huang, Jinhao and others},
  journal={arXiv preprint arXiv:2510.13999},
  year={2025}
}