zen5 — Mixture of Diverse Experts (MoDE)

zen5 is a next-generation AI architecture that routes across expert modules harvested from the largest open-source models using complexity-aware hierarchical routing.

Unlike standard models that always use the same compute, zen5 adapts compute to task difficulty:

Architecture

Input → ComplexityEstimator (64 tokens) → Tier Assignment
  T0: 0.8B dense → direct generation
  T1: 9B dense → standard generation
  T2: 10-40B MoE → multi-expert routing
  T3: 40-80B MoE → deep expert routing
  T4: 80-100B+ MoE → frontier expert routing + adaptive escalation

Key Innovations

1. Complexity-Aware Routing — Lightweight estimator predicts task difficulty from first 64 tokens, activates appropriate tier
2. Cross-Architecture Experts — Experts from 6+ diverse model families (Gated DeltaNet, Lightning Attention, DeepseekV3 MoE, FP8 MoE, GLM MoE)
3. MoE++ Zero-Computation Experts (ICLR 2025 Oral) — Zero/Copy/Constant experts for trivial tokens, 1.1-2.1x throughput
4. ReMoE ReLU Routing (ICLR 2025) — Adaptive expert count via ReLU gating (no fixed Top-K)
5. Adaptive Escalation — Mid-generation promotion to higher tiers when confidence drops
6. Transfusion — Unified autoregressive (text) + diffusion (3D/video) in single architecture

Expert Pool (~3.1T total parameters)

Omnimodal Experts

Model Lineup

Trainable Parameters

Only ~394M parameters are trained (0.013% of total model):

• ComplexityEstimator: 207M — predicts task difficulty tier
• AlignmentLayer: 134M — projects diverse architectures to shared space
• MoDERouter: 52M — per-tier expert routing with ReLU gating

All expert weights are frozen — extracted and served as-is from source models.

Quick Start

# List expert pool
python scripts/expert_extraction.py list

# Extract experts from source models
python scripts/expert_extraction.py extract-all --output ./experts/

# Train complexity router
python scripts/router_training.py train --output ./checkpoints/

# Benchmark routing accuracy
python scripts/benchmark.py routing --checkpoint ./checkpoints/router/best.pt

# Generate full evaluation report
python scripts/benchmark.py report --checkpoint ./checkpoints/router/best.pt

Training

zen5 is trained in the open on the Hanzo Network. Training consists of 5 phases:

1. Expert Extraction — Harvest FFN/MoE blocks from source models
2. Alignment Pre-training — Learn cross-architecture projections
3. Router Training — Train complexity estimator and tier routers
4. Integration — Joint fine-tuning with frozen experts
5. Omnimodal Fusion — Add vision/video/3D/audio experts

See docs/ARCHITECTURE.md for detailed training strategy and hardware requirements.

Repository Structure

zen5/
├── scripts/
│   ├── expert_extraction.py   # Extract experts from source models
│   ├── router_training.py     # Train complexity router + alignment
│   ├── benchmark.py           # Evaluation and benchmarking suite
│   └── hf_repos.py           # HuggingFace model card generation
├── docs/
│   └── ARCHITECTURE.md        # Full architecture specification
├── training/                  # Training configs and data
├── checkpoints/               # Model checkpoints
└── paper/                     # Technical report

Requirements

torch>=2.4
transformers>=4.48
safetensors
huggingface_hub

References

• MoE++ (ICLR 2025 Oral) — Zero-Computation Experts
• ReMoE (ICLR 2025) — ReLU-based MoE routing
• HMoE (EMNLP 2025) — Heterogeneous expert sizes
• Symbolic-MoE (2025) — Skill-based routing
• Uni-MoE-2.0 — Dynamic capacity multimodal MoE
• Transfusion (Meta 2024) — Unified AR + diffusion

Links

• Zen LM — Model family home
• Hanzo AI — Infrastructure and training
• HuggingFace — Model downloads
• Architecture Paper — Coming soon

License

Apache 2.0

---

zen5: Mixture of Diverse Experts — Clarity Through Diversity