🌟 AuroraAI Router

Intelligent on-device LLM routing for mobile applications with Cactus Compute

AuroraAI Router is a lightweight, mobile-optimized model selection system that intelligently routes prompts to the best available LLM on your device. Based on DeepMind's UniRouter approach, it uses cluster-based routing with per-cluster error rates to balance quality and performance.

---

🎯 Features

• ✅ 100% On-Device - No API calls, guaranteed privacy
• ⚡ Ultra-Fast Routing - <20ms decision time
• 📱 Mobile-Optimized - <5MB profile size, minimal memory footprint
• 🎓 ML-Powered - Cluster-based selection using semantic embeddings
• 🔧 Cactus Integration - Native support for Cactus Compute models
• 🌐 Cross-Platform - Python, C++, Kotlin, Flutter SDKs

---

📊 How It Works

User Prompt
    ↓
1. Extract embedding (SentenceTransformers)
    ↓
2. Assign to cluster (K-means, <5ms)
    ↓
3. Score models: score = error_rate[cluster] + λ × normalized_size
    ↓
4. Select best model
    ↓
Load & Run Model (Cactus)

Example Routing Decisions

---

🚀 Quick Start

Installation

cd auroraai-router
pip install -r requirements.txt
pip install -e .  # Editable install

Basic Usage (Python)

from auroraai_router import AuroraAIRouter, ModelInfo

# Define your Cactus models
models = [
    ModelInfo(
        model_id='gemma-270m',
        model_path='weights/gemma-3-270m-it',
        size_mb=172,
        avg_tokens_per_sec=173
    ),
    ModelInfo(
        model_id='qwen-1.7b',
        model_path='weights/Qwen3-1.7B',
        size_mb=1161,
        avg_tokens_per_sec=75
    ),
]

# Initialize router
router = AuroraAIRouter(
    profile_path='profiles/cactus_models_profile.json',
    models=models
)

# Route a prompt
result = router.route(
    prompt="Explain how neural networks work",
    cost_preference=0.7  # 0=fast, 1=quality
)

print(f"Selected: {result.model_id}")
print(f"Model path: {result.model_path}")
print(f"Estimated latency: {result.estimated_latency_ms:.0f}ms")

# Use with Cactus (pseudocode)
# model = cactus_init(result.model_path, 2048)
# response = cactus_complete(model, messages, ...)

---

📚 Creating Router Profiles

Use the provided Jupyter notebooks to create profiles for your models:

1. Profile Cactus Models

jupyter notebook notebooks/01_profile_cactus_models.ipynb

This notebook:

• Loads benchmark datasets
• Runs inference on all Cactus models (or simulates)
• Creates clusters based on prompt similarity
• Computes per-cluster error rates
• Saves optimized router profile

2. Test Router

jupyter notebook notebooks/02_test_routing.ipynb

This notebook:

• Loads the router profile
• Tests routing decisions
• Benchmarks performance
• Visualizes routing behavior

---

📖 Project Structure

auroraai-router/
├── core/                         # Core Python library
│   ├── mobile_cluster_engine.py  # Lightweight clustering
│   ├── mobile_router.py          # Router logic
│   └── profile_converter.py      # Profile utilities
│
├── notebooks/                    # Jupyter notebooks
│   ├── 01_profile_cactus_models.ipynb
│   └── 02_test_routing.ipynb
│
├── profiles/                     # Router profiles
│   └── cactus_models_profile.json
│
├── sdks/                         # Language SDKs
│   ├── python/                   # Python SDK
│   ├── kotlin/                   # Android/Kotlin (TODO)
│   └── flutter/                  # Flutter/Dart (TODO)
│
├── router-native/                # C++ implementation
│   ├── include/cactus_router.h
│   └── src/router_core.cpp
│
├── examples/                     # Example code
│   ├── python/example_basic.py
│   ├── android/                  # Android example (TODO)
│   └── flutter/                  # Flutter example (TODO)
│
├── tests/                        # Unit tests
│   └── test_mobile_router.py
│
├── docs/                         # Documentation
│   └── API.md
│
├── requirements.txt
├── setup.py
└── README.md

---

🧪 Testing

Run Unit Tests

python tests/test_mobile_router.py

Or with pytest:

pytest tests/ -v

Run Example

python examples/python/example_basic.py

---

🔧 Advanced Usage

Custom Model Profiles

from core import ProfileConverter
import numpy as np

# Define your models
models = [
    {'model_id': 'my-model', 'size_mb': 300, 'avg_tokens_per_sec': 120}
]

# Define error rates (from your evaluation)
error_rates = {
    'my-model': [0.10, 0.12, 0.11, 0.09, 0.13]  # Per-cluster rates
}

# Create cluster centers (from your embeddings)
cluster_centers = np.random.randn(5, 384).astype(np.float32)

# Create profile
profile = ProfileConverter.create_cactus_profile(
    models_info=models,
    error_rates=error_rates,
    cluster_centers=cluster_centers,
    output_path='my_profile.json'
)

Convert Existing Router Profile

from core import ProfileConverter

# Convert from adaptive_router-main format
ProfileConverter.convert_to_mobile(
    source_profile_path='../adaptive_router-main/profile.json',
    output_path='profiles/mobile_profile.json',
    use_float16=True  # Reduce size
)

---

📐 Performance Metrics

Tested on: Pixel 6a, iPhone 13, Galaxy S21

---

🎓 How to Create Model Profiles

Step 1: Prepare Dataset

Create a CSV/JSON with columns: input, expected_output

input,expected_output
"What is 2+2?","4"
"Explain gravity","Gravity is a force..."

Step 2: Run Profiling Notebook

Open notebooks/01_profile_cactus_models.ipynb and:

1. Load your dataset
2. Run inference on all models (use Cactus)
3. Compute clusters and error rates
4. Save profile

Step 3: Test Router

Use notebooks/02_test_routing.ipynb to validate routing decisions.

---

🛠️ Integration with Cactus

Python + Cactus (Pseudocode)

from auroraai_router import AuroraAIRouter
# import cactus  # Your Cactus Python bindings

router = AuroraAIRouter('profile.json', models)

# Route prompt
result = router.route("Explain AI", cost_preference=0.6)

# Load selected model with Cactus
# model = cactus.init(result.model_path, 2048)
# response = model.complete([
#     {"role": "user", "content": "Explain AI"}
# ])
# print(response)

C++ + Cactus (Native)

#include "cactus.h"
#include "cactus_router.h"

// Initialize router
CactusRouterOptions opts = {
    .profile_path = "profile.json",
    .lambda_min = 0.0,
    .lambda_max = 2.0,
    .default_cost_preference = 0.5
};
CactusRouterHandle* router = cactus_router_init(&opts);

// Route prompt
CactusModelRecommendation result;
cactus_router_select(
    router,
    "Explain quantum physics",
    NULL,  // Auto-compute embedding
    0,
    NULL,  // All models
    0,
    0.8,   // Prefer quality
    &result
);

// Load and run model with Cactus
cactus_model_t model = cactus_init(result.model_path, 2048, NULL);
// ... use model ...

cactus_router_destroy(router);

---

📊 Cost Preference Guide

The cost_preference parameter (0.0 to 1.0) controls the quality-speed tradeoff:

• 0.0 - 0.3: Prefer small/fast models (Gemma-270m, SmolLM-360m)

  • Use for: Greetings, simple Q&A, quick tasks
  • Battery impact: Minimal
  • Latency: 100-300ms for 50 tokens

• 0.4 - 0.6: Balanced (Qwen-600m, LFM2-700M)

  • Use for: General tasks, moderate complexity
  • Battery impact: Low
  • Latency: 300-600ms for 50 tokens

• 0.7 - 1.0: Prefer quality (Qwen-1.7B)

  • Use for: Complex reasoning, coding, detailed answers
  • Battery impact: Moderate
  • Latency: 600-1200ms for 50 tokens

---

🔬 Based on Research

This router is based on:

1. UniRouter (DeepMind, 2025)

   • Paper: Universal Model Routing for Efficient LLM Inference
   • Cluster-based routing with feature vectors
   • Dynamic routing to unseen models

2. Cactus Compute

   • GitHub: cactus-compute/cactus
   • On-device AI inference engine
   • Optimized for mobile ARM CPUs

---

📝 TODO / Roadmap

• Python core library
• Profile converter
• Jupyter notebooks for profiling
• Python SDK
• Unit tests
• C++ router header (stub)
• Complete C++ implementation with JSON parsing
• Android/Kotlin SDK
• Flutter/Dart SDK
• iOS/Swift integration
• TF-IDF fallback (no embedding model required)
• ONNX Runtime integration for C++ embeddings
• Pre-trained profiles for common Cactus models
• Benchmarking suite

---

🤝 Contributing

Contributions welcome! This is an open research project.

1. Fork the repo
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a PR

---

📄 License

MIT License - see LICENSE file

---

🙏 Acknowledgments

• DeepMind - UniRouter research
• Cactus Compute - On-device inference engine
• HuggingFace - SentenceTransformers library
• Anthropic - Claude for development assistance

---

📞 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions

---

Built with ❤️ for the mobile AI community