Resources for talk at KubeCon + CloudNativeCon Europe 2026 Amsterdam: Intelligent Routing for Optimized Inference
Rust
4
31 commits
updated Mar 20, 2026
README
kubecon-eu-2026-agentic-routing
Resources for talk at KubeCon + CloudNativeCon Europe 2026 Amsterdam: Intelligent Routing for Optimized Inference
Overview
This lab shows how the intelligent router classifies the domain of every LLM request and automatically selects the right model from the pool. The client sends a plain request with no routing hint; AgentGateway calls the intelligent router via ExtProc (pre-routing), which reads the body, detects the domain, and injects the x-router-selected-model header. AgentGateway then selects the HTTPRoute based on that header.
Client request (no routing header)
│
▼
AgentGateway (port 80)
│ gateway-level ExtProc (PreRouting) ──► Intelligent Router (:18080)
│ reads body → classifies domain
│ injects x-router-selected-model: gpt-4.1
│ select_best_route() → HTTPRoute matches header
├─ gpt-4.1 → gpt-4-1 backend (finance / health / legal)
├─ gpt-5-mini → gpt-5-mini backend (science)
└─ gpt-4.1-mini → gpt-4-1-mini backend (technology / general)
▼
OpenAI API
The routing decision is proven two ways:
- Response
modelfield — OpenAI echoes back the actual model used, confirming which backend served the request. - Router logs —
[ext_proc] domain=finance selected_model=gpt-4.1shows the intelligent router classified the prompt and injected the routing header.
Building the router image: see artifacts/intelligent-router/README.md. The lab uses the pre-built image
antonioberben/intelligent-router:latest.
Prerequisites
| Tool | Install |
|---|---|
| Docker Desktop (with buildx) | docs.docker.com |
| kubectl ≥ 1.28 | brew install kubectl |
| helm ≥ 3.14 | brew install helm |
| jq | brew install jq |
| OpenAI API key | platform.openai.com |
| Existing Kubernetes cluster | Must be running and accessible via kubectl |
kubectl version --client && helm version --short && jq --version
Steps
And yes, you can install the whole application by running:
export OPENAI_API_KEY="sk-…"
export HF_TOKEN="hf_…"
curl -sL https://raw.githubusercontent.com/agentic-world-lab/kubecon-eu-2026-agentic-routing/main/install.sh | bash
If you want to go step by step, you can follow the guide here: Lab
Future Work: Advanced Hybrid LLM Routing Strategies
For a comprehensive deep-dive, see Hybrid LLM Routing Strategies.
The intelligent router demonstrated in this lab implements domain-aware quality/latency/cost scoring with dynamic budget pressure. This is one strategy among a rich and rapidly evolving taxonomy of hybrid routing techniques. Below is a curated summary of advanced strategies identified in contemporary research, grouped by category, that represent exciting future directions for this project.
Micro-Architectural & Token-Level Routing
| Strategy | Description |
|---|---|
| Token-Level Collaborative Routing (CITER) | A lightweight RL-trained router analyzes hidden states at every decode step, transferring execution to the cloud only when the local model's confidence collapses on a specific token. [44] |
| Adaptive Multi-Level Speculative Chain (SpecRouter) | Dynamically constructs an optimal chain of draft/verifier models on the fly based on real-time latency and token distribution divergence. [49] |
| MoE Dynamic Expert Capacity Budgeting | Restricts the number of active Mixture-of-Experts invoked during speculative decoding to prevent memory bandwidth starvation, increasing throughput by ~30%. [53] |
| Remaining-Token Orthogonality Pruning | Predicts sequence completion via token-to-sink attention analysis, pruning uninformative tokens to collapse complexity from quadratic to linear. [68] |
Memory & Cache-Aware Routing
| Strategy | Description |
|---|---|
| KV Cache-Affinity Optimization | Routes queries to the node already holding the matching prompt prefix in its attention cache, achieving >87% cache hit rates. [9] |
| PCIe Overlap I/O Optimization (KVPR) | Overlaps GPU recomputation with PCIe cache transfer to completely mask physical transfer latency. [36] |
| Dynamic VRAM Footprint Projection | Projects the total memory requirement of the context window, preemptively offloading to the cloud if the local node lacks contiguous VRAM. [27] |
Uncertainty & Quality Assurance
| Strategy | Description |
|---|---|
| Semantic Entropy Hallucination Routing | Forces multiple quantized local drafts, clusters them by meaning; high semantic divergence triggers reroute to a more capable cloud model. [56] |
| Semantic Energy Boltzmann Analysis | Applies a Boltzmann-inspired energy distribution to detect subtle reasoning failures more accurately than raw entropy. [60] |
| Consensus-Based Hierarchical Deflection | Duplicates high-stakes queries across heterogeneous cloud providers; a local judge model enforces majority-vote factual alignment. [61] |
| Hard-Blocking Long-Tail Filtration | A lightweight firewall model blocks unsolvable "long-tail" queries from consuming cloud compute, returning curated fallbacks. [82] |
Infrastructure & Hardware-Aware Routing
| Strategy | Description |
|---|---|
| Thermal and Power-Aware Scheduling (TAPAS) | Monitors die temperatures and fan telemetry, diverting prefill requests away from hardware approaching thermal throttling limits. [10] |
| Spot-Instance Volatility Arbitrage | Routes async bulk workloads to discounted, ephemeral cloud nodes; auto-fails back to local edge hardware on preemption events (~44% cost reduction). [11] |
| Network-Aware QoS Arbitration (SONAR) | Analyzes real-time packet loss and WAN latency, confining latency-sensitive payloads to local edge when trans-oceanic links degrade. [70] |
| Energy-per-Token Ecological Optimization | Calculates fluctuating electrical costs of local vs. cloud, routing to the endpoint with the lowest carbon footprint per token. [8] |
Security, Privacy & Compliance
| Strategy | Description |
|---|---|
| PII Vaulting and Format-Preservation | Routes through a local NER model to mask sensitive data with reversible vault identifiers before sending sterilized payloads to cloud APIs. [74] |
| Adversarial-Aware Deflection Sandbox | Scans prompts for jailbreak vectors via spatial-aware alignment, routing malicious payloads to isolated, read-only local logging models. [77] |
| Geo-Location Data Residency Constraints (GDPR) | Interrogates origin IP metadata to confine EU citizen data to EU-certified cloud zones or local edge clusters. [12] |
Economic & Operational Routing
| Strategy | Description |
|---|---|
| Real-Time Token Budget Degradation | Tracks aggregate tenant expenditure, automatically degrading routing from premium to free local models when financial limits are breached. [87] |
| Circuit-Breaker Auto-Ejection | Tracks continuous failure latency of external APIs, temporarily ejecting degraded endpoints from the routing table. [5] |
| A/B Testing Silent Traffic Mirroring | Duplicates a percentage of live cloud-bound traffic, silently routing the copy to a new local SLM to evaluate quality harmlessly. [98] |
| Policy Hot-Reloading Configuration | Extracts routing logic from external declarative configs, permitting live runtime updates without daemon restarts. [101] |
Key References
| # | Title | Link |
|---|---|---|
| 1 | Multi-LLM Routing Strategies for GenAI on AWS | aws.amazon.com |
| 5 | The Complete Guide to LLM Routing: 5 AI Gateways | medium.com |
| 9 | KV Cache Aware Routing with llm-d (Red Hat) | developers.redhat.com |
| 10 | TAPAS: Thermal- and Power-Aware Scheduling | arxiv.org |
| 11 | SkyServe: Serving AI Models with Spot Instances | arxiv.org |
| 12 | LLM-Driven Privacy-Aware Orchestration Across Cloud-Edge | arxiv.org |
| 44 | CITER: Token-Level Collaborative Routing | github.com |
| 49 | SpecRouter: Adaptive Multi-Level Speculative Decoding | arxiv.org |
| 53 | MoE-Spec: Expert Budgeting for Speculative Decoding | arxiv.org |
| 56 | Uncertainty-Based On-device LLM Routing | arxiv.org |
| 60 | Semantic Energy: Detecting Hallucination Beyond Entropy | arxiv.org |
| 74 | Privacy Gatekeepers for Cloud-Based AI Interactions | arxiv.org |
| 82 | Firewall Routing: Blocking for Better Hybrid Inference | aclanthology.org |
| 101 | NVIDIA AI Blueprint for Cost-Efficient LLM Routing | developer.nvidia.com |
Contributors
felipevicens
18 commits
antonioberben
13 commits
Similar repos
Languages
Rust
75.5%
Go
15.6%
Python
7.2%
Resources for talk at KubeCon + CloudNativeCon Europe 2026 Amsterdam: Intelligent Routing for Optimized Inference
Rust
4
31 commits
updated Mar 20, 2026
README
kubecon-eu-2026-agentic-routing
Resources for talk at KubeCon + CloudNativeCon Europe 2026 Amsterdam: Intelligent Routing for Optimized Inference
Overview
This lab shows how the intelligent router classifies the domain of every LLM request and automatically selects the right model from the pool. The client sends a plain request with no routing hint; AgentGateway calls the intelligent router via ExtProc (pre-routing), which reads the body, detects the domain, and injects the x-router-selected-model header. AgentGateway then selects the HTTPRoute based on that header.
Client request (no routing header)
│
▼
AgentGateway (port 80)
│ gateway-level ExtProc (PreRouting) ──► Intelligent Router (:18080)
│ reads body → classifies domain
│ injects x-router-selected-model: gpt-4.1
│ select_best_route() → HTTPRoute matches header
├─ gpt-4.1 → gpt-4-1 backend (finance / health / legal)
├─ gpt-5-mini → gpt-5-mini backend (science)
└─ gpt-4.1-mini → gpt-4-1-mini backend (technology / general)
▼
OpenAI API
The routing decision is proven two ways:
- Response
modelfield — OpenAI echoes back the actual model used, confirming which backend served the request. - Router logs —
[ext_proc] domain=finance selected_model=gpt-4.1shows the intelligent router classified the prompt and injected the routing header.
Building the router image: see artifacts/intelligent-router/README.md. The lab uses the pre-built image
antonioberben/intelligent-router:latest.
Prerequisites
| Tool | Install |
|---|---|
| Docker Desktop (with buildx) | docs.docker.com |
| kubectl ≥ 1.28 | brew install kubectl |
| helm ≥ 3.14 | brew install helm |
| jq | brew install jq |
| OpenAI API key | platform.openai.com |
| Existing Kubernetes cluster | Must be running and accessible via kubectl |
kubectl version --client && helm version --short && jq --version
Steps
And yes, you can install the whole application by running:
export OPENAI_API_KEY="sk-…"
export HF_TOKEN="hf_…"
curl -sL https://raw.githubusercontent.com/agentic-world-lab/kubecon-eu-2026-agentic-routing/main/install.sh | bash
If you want to go step by step, you can follow the guide here: Lab
Future Work: Advanced Hybrid LLM Routing Strategies
For a comprehensive deep-dive, see Hybrid LLM Routing Strategies.
The intelligent router demonstrated in this lab implements domain-aware quality/latency/cost scoring with dynamic budget pressure. This is one strategy among a rich and rapidly evolving taxonomy of hybrid routing techniques. Below is a curated summary of advanced strategies identified in contemporary research, grouped by category, that represent exciting future directions for this project.
Micro-Architectural & Token-Level Routing
| Strategy | Description |
|---|---|
| Token-Level Collaborative Routing (CITER) | A lightweight RL-trained router analyzes hidden states at every decode step, transferring execution to the cloud only when the local model's confidence collapses on a specific token. [44] |
| Adaptive Multi-Level Speculative Chain (SpecRouter) | Dynamically constructs an optimal chain of draft/verifier models on the fly based on real-time latency and token distribution divergence. [49] |
| MoE Dynamic Expert Capacity Budgeting | Restricts the number of active Mixture-of-Experts invoked during speculative decoding to prevent memory bandwidth starvation, increasing throughput by ~30%. [53] |
| Remaining-Token Orthogonality Pruning | Predicts sequence completion via token-to-sink attention analysis, pruning uninformative tokens to collapse complexity from quadratic to linear. [68] |
Memory & Cache-Aware Routing
| Strategy | Description |
|---|---|
| KV Cache-Affinity Optimization | Routes queries to the node already holding the matching prompt prefix in its attention cache, achieving >87% cache hit rates. [9] |
| PCIe Overlap I/O Optimization (KVPR) | Overlaps GPU recomputation with PCIe cache transfer to completely mask physical transfer latency. [36] |
| Dynamic VRAM Footprint Projection | Projects the total memory requirement of the context window, preemptively offloading to the cloud if the local node lacks contiguous VRAM. [27] |
Uncertainty & Quality Assurance
| Strategy | Description |
|---|---|
| Semantic Entropy Hallucination Routing | Forces multiple quantized local drafts, clusters them by meaning; high semantic divergence triggers reroute to a more capable cloud model. [56] |
| Semantic Energy Boltzmann Analysis | Applies a Boltzmann-inspired energy distribution to detect subtle reasoning failures more accurately than raw entropy. [60] |
| Consensus-Based Hierarchical Deflection | Duplicates high-stakes queries across heterogeneous cloud providers; a local judge model enforces majority-vote factual alignment. [61] |
| Hard-Blocking Long-Tail Filtration | A lightweight firewall model blocks unsolvable "long-tail" queries from consuming cloud compute, returning curated fallbacks. [82] |
Infrastructure & Hardware-Aware Routing
| Strategy | Description |
|---|---|
| Thermal and Power-Aware Scheduling (TAPAS) | Monitors die temperatures and fan telemetry, diverting prefill requests away from hardware approaching thermal throttling limits. [10] |
| Spot-Instance Volatility Arbitrage | Routes async bulk workloads to discounted, ephemeral cloud nodes; auto-fails back to local edge hardware on preemption events (~44% cost reduction). [11] |
| Network-Aware QoS Arbitration (SONAR) | Analyzes real-time packet loss and WAN latency, confining latency-sensitive payloads to local edge when trans-oceanic links degrade. [70] |
| Energy-per-Token Ecological Optimization | Calculates fluctuating electrical costs of local vs. cloud, routing to the endpoint with the lowest carbon footprint per token. [8] |
Security, Privacy & Compliance
| Strategy | Description |
|---|---|
| PII Vaulting and Format-Preservation | Routes through a local NER model to mask sensitive data with reversible vault identifiers before sending sterilized payloads to cloud APIs. [74] |
| Adversarial-Aware Deflection Sandbox | Scans prompts for jailbreak vectors via spatial-aware alignment, routing malicious payloads to isolated, read-only local logging models. [77] |
| Geo-Location Data Residency Constraints (GDPR) | Interrogates origin IP metadata to confine EU citizen data to EU-certified cloud zones or local edge clusters. [12] |
Economic & Operational Routing
| Strategy | Description |
|---|---|
| Real-Time Token Budget Degradation | Tracks aggregate tenant expenditure, automatically degrading routing from premium to free local models when financial limits are breached. [87] |
| Circuit-Breaker Auto-Ejection | Tracks continuous failure latency of external APIs, temporarily ejecting degraded endpoints from the routing table. [5] |
| A/B Testing Silent Traffic Mirroring | Duplicates a percentage of live cloud-bound traffic, silently routing the copy to a new local SLM to evaluate quality harmlessly. [98] |
| Policy Hot-Reloading Configuration | Extracts routing logic from external declarative configs, permitting live runtime updates without daemon restarts. [101] |
Key References
| # | Title | Link |
|---|---|---|
| 1 | Multi-LLM Routing Strategies for GenAI on AWS | aws.amazon.com |
| 5 | The Complete Guide to LLM Routing: 5 AI Gateways | medium.com |
| 9 | KV Cache Aware Routing with llm-d (Red Hat) | developers.redhat.com |
| 10 | TAPAS: Thermal- and Power-Aware Scheduling | arxiv.org |
| 11 | SkyServe: Serving AI Models with Spot Instances | arxiv.org |
| 12 | LLM-Driven Privacy-Aware Orchestration Across Cloud-Edge | arxiv.org |
| 44 | CITER: Token-Level Collaborative Routing | github.com |
| 49 | SpecRouter: Adaptive Multi-Level Speculative Decoding | arxiv.org |
| 53 | MoE-Spec: Expert Budgeting for Speculative Decoding | arxiv.org |
| 56 | Uncertainty-Based On-device LLM Routing | arxiv.org |
| 60 | Semantic Energy: Detecting Hallucination Beyond Entropy | arxiv.org |
| 74 | Privacy Gatekeepers for Cloud-Based AI Interactions | arxiv.org |
| 82 | Firewall Routing: Blocking for Better Hybrid Inference | aclanthology.org |
| 101 | NVIDIA AI Blueprint for Cost-Efficient LLM Routing | developer.nvidia.com |
Similar repos
Contributors
felipevicens
18 commits
antonioberben
13 commits
Languages
Rust
75.5%
Go
15.6%
Python
7.2%