YOLO-Master: MOE-Accelerated with Specialized Transformers for Enhanced Real-time Detection.

Xu Lin^1*, Jinlong Peng^1*, Zhenye Gan¹, Jiawen Zhu², Jun Liu^1
1Tencent Youtu Lab    ²Singapore Management University
^*Equal Contribution

YOLO-Master is a YOLO-style framework tailored for Real-Time Object Detection (RTOD). It marks the first deep integration of Mixture-of-Experts (MoE) into the YOLO architecture for general datasets. By leveraging Efficient Sparse MoE (ES-MoE) and lightweight Dynamic Routing, the framework achieves instance-conditional adaptive computation. This "compute-on-demand" paradigm allows the model to allocate FLOPs based on scene complexity, reaching a superior Pareto frontier between high precision and ultra-low latency.

Key Highlights:

• Methodological Innovation (ES-MoE + Dynamic Routing): Utilizes dynamic routing networks to guide expert specialization during training and activates only the most relevant experts during inference, significantly reducing redundant computation while boosting detection performance.
• Performance Validated (Accuracy × Latency): On MS COCO, YOLO-Master-N achieves 42.4% AP @ 1.62ms latency, outperforming YOLOv13-N with a +0.8% mAP gain while being 17.8% faster.
• Compute-on-Demand Intuition: Transitions from "static dense computation" to "input-adaptive compute allocation," yielding more pronounced gains in dense or challenging scenarios.
• Out-of-the-Box Pipeline: Provides a complete end-to-end workflow including installation, validation, training, inference, and deployment (ONNX, TensorRT, etc.).
• Continuous Engineering Evolution: Includes advanced utilities such as MoE pruning and diagnostic tools (diagnose_model / prune_moe_model), CW-NMS, and Sparse SAHI inference modes.

💡 A Humble Beginning (Introduction)

"Exploring the frontiers of Dynamic Intelligence in YOLO."

This work represents our passionate exploration into the evolution of Real-Time Object Detection (RTOD). To the best of our knowledge, YOLO-Master is the first work to deeply integrate Mixture-of-Experts (MoE) with the YOLO architecture on general-purpose datasets.

Most existing YOLO models rely on static, dense computation—allocating the same computational budget to a simple sky background as they do to a complex, crowded intersection. We believe detection models should be more "adaptive", much like the human visual system. While this initial exploration may be not perfect, it demonstrates the significant potential of Efficient Sparse MoE (ES-MoE) in balancing high precision with ultra-low latency. We are committed to continuous iteration and optimization to refine this approach further.

Looking forward, we draw inspiration from the transformative advancements in LLMs and VLMs. We are committed to refining this approach and extending these insights to fundamental vision tasks, with the ultimate goal of tackling more ambitious frontiers like Open-Vocabulary Detection and Open-Set Segmentation.

🎨 Architecture

YOLO-Master introduces ES-MoE blocks to achieve "compute-on-demand" via dynamic routing.

📚 In-Depth Documentation

For a deep dive into the design philosophy of MoE modules, detailed routing mechanisms, and optimization guides for deployment on various hardware (GPU/CPU/NPU), please refer to our Wiki: 👉 Wiki: MoE Modules Explained

📖 Table of Contents

• A Humble Beginning
• Architecture
• Updates
• New Features (v2026.02)
  • Mixture of Experts (MoE)
  • LoRA Fine-Tuning
  • Sparse SAHI
  • Cluster-Weighted NMS
  • Mixture-of-Attention (MoA)
  • Mixture-of-Transformers (MoT)
  • Agent Skill System
• Main Results
  • Detection
  • Segmentation
  • Classification
• Model Zoo
• Detection Examples
• Supported Tasks
• Quick Start
  • Installation
  • Validation
  • Training
  • Inference
  • Export
  • Gradio Demo
• Community & Contributing
• License
• Acknowledgements
• Citation

🚀 Updates (Latest First)

• 2026/08/07: 📄 New paper: YOLO-PEFT: Parameter-Efficient Fine-Tuning on YOLO Family (PDF) — proposes a structure-aware, auditable constraint-planning framework for PEFT placement in YOLO detectors. Within the evaluated coverage, planner-selected RS-LoRA reaches 0.7138 / 0.7307 mAP50-95 on YOLO11s / YOLO12s versus 0.6428 / 0.6662 for Full-SFT; all seven evaluated LoRA-family configurations on RT-DETR-L trigger a calibrated Refuse-to-Full-SFT decision. A controlled YOLO11 audit reports 43.9% lower peak training memory with 1.72x training time. Frozen claim table: reports/yolo-peft-2608.07051-anchor.json; verify with python scripts/reproduce_yolo_peft_paper.py --check-anchor.
• 2026-06-29: 🤖✅ Agent Skill System Validation Complete — Full end-to-end validation of yolo-master-agent Skill with 50/50 test cases passing across 8 suites (quick / fast-smoke / cli-smoke / dry-run / contract / deep-smoke / extended / all). Fixed AttributeError from missing end2end field in default.yaml. Verified complete training → validation → inference pipeline with MPS auto-selection and workers=0 auto-completion. Skill runners: yolo.train, yolo.val, yolo.predict, yolo.benchmark, yolo.export, yolo.lora.diagnose, yolo.eval.peft_compare, yolo.multimodal.infer, yolo.system.doctor.
• 2026-06-29: 🦏🏆 Selected for Tencent Rhino Bird Open Source Program 2026 — YOLO-Master has been officially selected for the Tencent Rhino Bird Open Source Program (Summer of Code 2026). This program aims to cultivate outstanding open-source talents and promote the prosperity and development of the open-source community. YOLO-Master will receive continuous support from the Tencent Open Source Fund, including mentorship, resource allocation, and community promotion, to further advance the integration of dynamic intelligence and MoE architecture in real-time object detection.
• 2026-06-28: 🔀 MoA + MoT Integration — Mixture-of-Attention (MoA) and Mixture-of-Transformers (MoT) modules merged into main with regression tests. MoA: lightweight router assigns tokens to attention heads with different receptive fields (Local / Regional / Global). MoT: content-aware router assigns tokens to distinct Transformer experts (LocalConvTransformer / WindowTransformer / DeformableTransformer) with soft Top-K blending and optional load-balancing aux loss. New model configs added under ultralytics/cfg/models/master/v0_1/.
• 2026-06-25: 🧠 MoA Module Introduced — Mixture-of-Attention (MoA) for CNN-native multi-scale attention fusion. 1×1 conv router assigns soft probabilities to Local (depthwise-3×3), Regional (pooled-stride=2), and Global (linear-attention) heads. Zero seq-dim reshape, Flash-Attention compatible. Supports C2fMoA wrapper and NeckMoAFusion cross-scale FPN/PAN fusion.
• 2026-06-25: 🔄 MoT Module Introduced — Mixture-of-Transformers (MoT) routing module. Three complete Transformer expert architectures with soft Top-K blending, static-graph trace stability for ONNX/TorchScript, and optional z-loss style load-balancing aux loss.
• 2026-05-12: 🤖 Agent Skill System Introduced — yolo-master-agent Skill Bundle with SKILL.md, autotrain validation suite (50+ cases), multimodal evaluation, async evaluation, MPS auto-selection, dry-run/contract/smoke tiered validation, and VLM/LLM cooperative inference (OpenAI / DashScope compatible).
• 2026/02/21: 🎉🎉 Our paper has been accepted by CVPR 2026! Thank you to all the contributors and community members for your support!
• 2026/02/13: 🧨🚀add LoRA support for model training and release v2026.02 version.[Happy New Year!]
• 2026/01/16: [feature] Add pruning and analysis tools for MoE models.

  1. diagnose_model: Visualize expert utilization and routing behavior to identify redundant experts.
  2. prune_moe_model: Physically excise redundant experts and reconstruct routers for efficient inference without retraining.

• 2026/01/16: Repo isLinXu/YOLO-Master transferred to Tencent.
• 2026/01/14: ncnn-YOLO-Master-android support deploy YOLO-Master. Thanks to them!
• 2026/01/09: [feature] Add Cluster-Weighted NMS (CW-NMS) to trade mAP vs speed.

  cluster: False # (bool) cluster NMS (MoE optimized)

• 2026/01/07: TensorRT-YOLO accelerates YOLO-Master. Thanks to them!
• 2026/01/07: Add MoE loss explicitly into training.

  Epoch GPU_mem box_loss cls_loss dfl_loss moe_loss Instances Size

• 2026/01/04: Split MoE script into modules

  Split MoE script into separate modules (routers, experts)

• 2026/01/03: [feature] Added Sparse SAHI Inference Mode: Introduced a content-adaptive sparse slicing mechanism guided by a global Objectness Mask, significantly accelerating small object detection in high-resolution images while optimizing GPU memory efficiency.
• 2025/12/31: Released the demo YOLO-Master-WebUI-Demo.
• 2025/12/31: Released YOLO-Master v0.1 with code, pre-trained weights, and documentation.
• 2025/12/30: arXiv paper published.

🔥 New Features (v2026.02)

1️⃣ Mixture of Experts (MoE) Support

YOLO-Master introduces the first deep integration of Mixture-of-Experts into the YOLO architecture, enabling instance-conditional adaptive computation.

Core Components:

Usage:

from ultralytics import YOLO

# Load MoE configuration
model = YOLO("ultralytics/cfg/models/master/v0_1/det/yolo-master-n.yaml")

# Training with MoE
results = model.train(
    data="coco8.yaml",
    epochs=100,
    imgsz=640,
    batch=16,
    moe_num_experts=8,      # Number of experts
    moe_top_k=2,            # Experts activated per token
    moe_balance_loss=0.01,  # Load balancing loss weight
)

# Expert utilization analysis & pruning
model.prune_experts(threshold=0.15)

💡 Looking for a plug-and-play ES-MoE module for all YOLO versions?

If you are seeking a convenient, pluggable ES-MoE module integration for the full YOLO family, please refer to the community library: Lfan-ke/ES-MoE, ES-MoE for PyPI.

2️⃣ LoRA Support - Parameter-Efficient Fine-Tuning

Architecture-agnostic LoRA adaptation with zero architectural overhead — enabled purely through configuration, no model surgery required.

LoRA vs Full SFT vs DoRA vs LoHa: Training curves comparison on YOLOv11-s (COCO val2017, 300 epochs)

Key Advantages:

• 🎯 Using ~10% trainable parameters to achieve 95-98% of full fine-tuning performance
• ⚡ In the controlled YOLO11 audit reported by YOLO-PEFT, peak training memory decreased by 43.9%, while training took 1.72× as long as Full-SFT. Results depend on the model, adapter configuration, data, and hardware; training speedup is not guaranteed. See the paper claim anchor for reproduction entry points.
• 📦 Ultra-compact adapters (e.g., YOLO11x: 14.1 MB adapter vs 114.6 MB full model)

Supported Models:

Usage:

from ultralytics import YOLO

model = YOLO("yolo11s.pt")

# LoRA training (one-click activation)
results = model.train(
    data="coco8.yaml",
    epochs=300,
    imgsz=640,
    batch=32,
    lora_r=16,                # rank=16, best cost-effectiveness
    lora_alpha=32,            # alpha = 2×r
    lora_dropout=0.1,
    lora_gradient_checkpointing=True,
)

# Save only LoRA adapters (~4.1MB for YOLO11s) to a directory
model.save_lora_only("yolo11s_lora_r16")

🚀 Downstream Scenario Low-Rank Adaptation (LoRA) Matrix Test Report

To verify the performance and resource consumption of the YOLO-Master LoRA framework across different downstream tasks, we conducted a cross-Rank matrix benchmark on the RTX 5070 Ti hardware platform using two typical scenarios (lightweight vs. extreme complexity).

* Note: The trainable parameter ratio is a theoretical estimate relative to the backbone's total parameters. Since Ultralytics automatically resets the gradient graph and triggers weight fusion after the training lifecycle ends, the runtime dynamic statistics remain silent.

** Selection Suggestions & Pitfalls to Avoid:**

1. Lightweight/Simple Scenarios (e.g., Brain Tumor): Rank = 4 is recommended. For datasets with single-feature profiles, lower Ranks provide better regularization and effectively mitigate overfitting.
   • Pitfall: Medical images (MRI/CT) are physically single-channel (grayscale), whereas YOLO defaults to 3-channel (RGB) inputs. Please ensure channel alignment in your data pipeline or use customized ch=1 convolution configurations.
2. Dense/Complex Scenarios (e.g., VisDrone): Rank >= 16 is recommended. Complex backgrounds and high-density tiny objects require larger parameter capacities for representation. Increasing the Rank significantly improves mAP50 by +25.4% (with <0.1 GB VRAM overhead).
   • Pitfall: Aerial imagery suffers from extreme scale variations. It is strictly advised to enable the built-in Sparse SAHI (Sparse Inference) or set rect: true during training. Furthermore, include both the Backbone's dimensionality reduction layers and MoE Experts' MLPs in lora_target_modules to provide sufficient degrees of freedom for spatial local features.

3️⃣ Sparse SAHI Mode

Sparse Slicing Aided Hyper-Inference — a revolutionary optimization for ultra-large image (4K/8K) detection, achieving 3-5x speedup by intelligently skipping blank regions.

Sparse SAHI pipeline: Objectness Mask → Adaptive Slicing → High-Resolution Inference → CW-NMS Merging

Left: Skip ratio analysis across different scenes. Right: Real-world detection example.

How it works:

1. 🗺️ Low-resolution full-image inference generates an objectness heatmap
2. ✂️ Adaptive slicing skips regions with objectness < 0.15
3. 🎯 High-resolution inference only on regions of interest
4. 🔗 Multi-slice results merged via CW-NMS

Usage:

from ultralytics import YOLO

model = YOLO("yolov8n.pt")

results = model.predict(
    source="large_aerial_image.jpg",
    sparse_sahi=True,
    slice_size=640,
    overlap_ratio=0.2,
    objectness_threshold=0.15,
)

4️⃣ Cluster-Weighted NMS (CW-NMS)

Cluster-based detection box fusion algorithm using Gaussian-weighted averaging instead of hard suppression, significantly improving localization accuracy.

CW-NMS vs Traditional NMS vs Soft-NMS: Performance comparison on dense scenes

from ultralytics import YOLO

model = YOLO("yolov8n.pt")
results = model.predict(
    source="dense_objects.jpg",
    cluster=True,     # Enable CW-NMS
    sigma=0.1,        # Gaussian weight σ
)

5️⃣ Mixture-of-Attention (MoA) Support

YOLO-Master introduces Mixture-of-Attention (MoA), a CNN-native multi-scale attention fusion mechanism. A lightweight 1×1 conv router assigns soft probabilities to each spatial token across three attention head groups with different receptive fields: Local (depthwise-3×3), Regional (pooled-stride=2), and Global (linear-attention). Zero sequence-dimension reshape, fully Flash-Attention compatible.

Key Features:

• 🧠 Multi-scale attention fusion: Local detail + Regional context + Global semantics in one block
• ⚡ CNN-native: [B,C,H,W] → [B,C,H,W], no seq-dim reshape, compatible with all YOLO backbones
• 🔗 Flexible integration: C2fMoA wrapper for direct C2f/C3k2 replacement; NeckMoAFusion for cross-scale FPN/PAN fusion

Usage:

from ultralytics import YOLO

# Load MoA configuration
model = YOLO("ultralytics/cfg/models/master/v0_1/det/yolo-master-n.yaml")

# Training with MoA
results = model.train(
    data="coco8.yaml",
    epochs=100,
    imgsz=640,
    batch=16,
)

6️⃣ Mixture-of-Transformers (MoT) Support

YOLO-Master introduces Mixture-of-Transformers (MoT), a content-aware routing module that distributes tokens to specialized Transformer experts.

Three Expert Architectures:

• 🏠 LocalConvTransformer: Depth-wise convolutions for texture and edge detection
• 🪟 WindowTransformer: Swin-style window partitioning for medium-scale objects
• 🌀 DeformableTransformer: Sparse deformable sampling for irregular and occluded objects

Key Features:

• 🧭 Soft Top-K Routing: Static-graph computes all experts, blends outputs via Top-K mask weights — ONNX/TorchScript trace stable
• ⚖️ Optional Load-Balancing: z-loss style auxiliary loss for stable expert utilization
• 📦 Out-of-the-Box: Compatible with existing YOLO-Master training and export pipelines

Usage:

from ultralytics import YOLO

# Load MoT configuration
model = YOLO("ultralytics/cfg/models/master/v0_1/det/yolo-master-n.yaml")

# Training with MoT
results = model.train(
    data="coco8.yaml",
    epochs=100,
    imgsz=640,
    batch=16,
)

7️⃣ Agent Skill System

YOLO-Master introduces the yolo-master-agent Skill Bundle, enabling AI agents to orchestrate training, validation, inference, and evaluation through a structured Skill interface.

Key Features:

• 🤖 9 Skill Runners: yolo.train, yolo.val, yolo.predict, yolo.benchmark, yolo.export, yolo.lora.diagnose, yolo.eval.peft_compare, yolo.multimodal.infer, yolo.system.doctor
• ✅ 50+ Test Cases: 8 validation suites (quick / fast-smoke / cli-smoke / dry-run / contract / deep-smoke / extended / all)
• 🧠 Multimodal Inference: YOLO detection + VLM/LLM cooperative reasoning, supporting OpenAI / DashScope compatible endpoints
• 🔧 Auto-Device: MPS/CPU/CUDA auto-selection, workers=0 auto-completion, safe defaults
• 📊 Structured Output: JSON manifest with training metrics, resource usage, error classification, and next-action recommendations

Usage:

# Via Python API
from ultralytics import YOLO
model = YOLO("yolo11n.pt")
results = model.train(data="coco8.yaml", epochs=1, imgsz=640)

# Via Skill CLI
python agent/scripts/run_yolo_master_skill.py \
    --json '{"skill":"yolo.train","inputs":{"model":"yolo11n.pt","data":"coco8.yaml"},"params":{"epochs":1,"imgsz":640}}'

📊 Main Results

Detection

Segmentation

Classification

📦 Model Zoo & Benchmarks

Pretrained .pt weights are synced from the YOLO-Master-v26.02 release.

YOLO-Master-EsMoE Series

YOLO-Master-v0.1 Series

🖼️ Detection Examples

Detection
Segmentation

🧩 Supported Tasks

YOLO-Master builds upon the robust Ultralytics framework, inheriting support for various computer vision tasks. While our research primarily focuses on Real-Time Object Detection, the codebase is capable of supporting:

⚙️ Quick Start

Installation

# 1. Create and activate a new environment
conda create -n yolo_master python=3.11 -y
conda activate yolo_master

# 2. Clone the repository
git clone https://github.com/Tencent/YOLO-Master
cd YOLO-Master

# 3. Install dependencies
pip install -r requirements.txt
pip install -e .

# 4. Optional: Install FlashAttention for faster training (CUDA required)
pip install flash_attn

Validation

Validate the model accuracy on the COCO dataset.

from ultralytics import YOLO

# Load the pretrained model
model = YOLO("yolo_master_n.pt")

# Run validation
metrics = model.val(data="coco.yaml", save_json=True)
print(metrics.box.map)  # map50-95

Training

Train a new model on your custom dataset or COCO.

from ultralytics import YOLO

# Load a model
model = YOLO('cfg/models/master/v0/det/yolo-master-n.yaml')  # build a new model from YAML

# Train the model
results = model.train(
    data='coco.yaml',
    epochs=600,
    batch=256,
    imgsz=640,
    device="0,1,2,3", # Use multiple GPUs
    scale=0.5,
    mosaic=1.0,
    mixup=0.0,
    copy_paste=0.1
)

Inference

Run inference on images or videos.

Python:

from ultralytics import YOLO

model = YOLO("yolo_master_n.pt")
results = model("path/to/image.jpg")
results[0].show()

CLI:

yolo predict model=yolo_master_n.pt source='path/to/image.jpg' show=True

Export

Export the model to other formats for deployment (TensorRT, ONNX, etc.).

from ultralytics import YOLO

model = YOLO("yolo_master_n.pt")
model.export(format="engine", half=True)  # Export to TensorRT
# formats: onnx, openvino, engine, coreml, saved_model, pb, tflite, edgetpu, tfjs

Gradio Demo

Launch a local web interface to test the model interactively. This application provides a user-friendly Gradio dashboard for model inference, supporting automatic model scanning, task switching (Detection, Segmentation, Classification), and real-time visualization.

python start_studio.py
# Open http://127.0.0.1:7860 in your browser

This recommended launcher starts both the FastAPI Studio Job API and the Gradio WebUI. For manual two-process startup, run python main_engine.py first and then python app.py in a second terminal. python app.py remains a Gradio-only entrypoint and requires the API configured by F1_STUDIO_API_URL (default: http://127.0.0.1:8000).

🤝 Community & Contributing

We welcome contributions! Please check out our Contribution Guidelines for details on how to get involved.

• Issues: Report bugs or request features here.
• Pull Requests: Submit your improvements.

📄 License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0).

🙏 Acknowledgements

This work builds upon the excellent Ultralytics framework. Huge thanks to the community for contributions, deployments, and tutorials!

📝 Citation

If you use YOLO-Master in your research, please cite our paper:

@inproceedings{lin2026yolomaster,
  title={{YOLO-Master}: MOE-Accelerated with Specialized Transformers for Enhanced Real-time Detection},
  author={Lin, Xu and Peng, Jinlong and Gan, Zhenye and Zhu, Jiawen and Liu, Jun},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2026}
}

For the parameter-efficient fine-tuning work, please also cite:

@misc{lin2026yolopeft,
  title={YOLO-PEFT: Parameter-Efficient Fine-Tuning on YOLO Family},
  author={Lin, Xu and Nie, WenJie and Peng, Jinlong and Fu, Weifu and Ma, YueXiao and Zheng, Xiawu and Liu, Yong},
  year={2026},
  eprint={2608.07051},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2608.07051}
}

⭐ If you find this work useful, please star the repository!