Adaptive Model Serving

Route every request to the smallest model that can handle it. A local three-model serving stack with a difficulty judge at the front door — built on a 70-prompt eval that falsified the project's original thesis and redirected the design halfway through.

Headline result: kept 98% of the largest model's quality at 42% less compute time, measured against a labeled eval set, not estimated.

---

The story (read this first)

Original thesis: serve one model at multiple quantization levels (4-bit / 8-bit / fp16) and route each request to the cheapest precision that can handle it.

What the data said: we built the eval first — 70 prompts across five categories with automatic grading. Result: at 1.5B parameters, precision made no measurable difference in quality on any category, while 4-bit was ~2x faster than fp16.

Conclusion: quantization is free at this scale. So we quantize everything to 4-bit and route across the axis where quality actually lives — model size:

A real quality ladder (the 0.5B model falls for 29 of 30 trick questions) and a 5x cost spread between rungs. Now routing has a job.

The router

Two judges compete to predict, per request, which rung is enough:

• judge-v0 — ~30 lines of heuristics: reasoning-shaped patterns ("all but", "how many ... left", "how is that possible"), story-numbers detection, math/code/summary keywords.
• judge-v1 — the 0.5B model itself classifies each prompt as easy/medium/hard (~0.08s overhead per request).

Policies are scored offline against logged eval outcomes — we already know which model passed which prompt and how long it took, so any routing policy's exact quality and cost can be computed without re-running models.

The regex heuristics beat the LLM judge. judge-v1 routed only 2 of 70 prompts to the large model and rated trick questions "easy" — the same trick questions it fails itself. A model too small to solve hard problems is also too small to recognize them. judge-v0's entire quality loss vs always-3B traces to a single under-routed hard-arithmetic prompt.

Eval methodology notes

• 70 prompts, 5 categories; reasoning traps are the discriminating category (30 prompts, each with a documented misdirection).
• Code answers are graded by executing the generated function against test cases.
• Trap prompts require a structured final line ("Answer: ...") and only that line is graded. This fixed two grader bugs found by failure analysis: correct conclusions restated in other units being mis-scored, and rambling non-conclusions being credited for mentioning the right word mid-stream. Audit your grader before trusting your eval.
• Known imperfections: n=30 traps gives ±coarse pass rates; one 3B summarization failure is a keyword-grading edge case.

The serving layer

FastAPI app that loads all three models at startup (all Q4_K_M, ~3.5 GB RAM total) and routes live requests through judge-v0:

• POST /ask — routes, answers, and returns metadata: chosen model, routing reason, latency, estimated time saved vs always-3B. Supports force_tier override for demos.
• GET /stats — running session mix and cumulative compute saved.

pip3 install -r requirements.txt
python3 -m uvicorn server.app:app --port 8000
# then open http://localhost:8000/docs

Repo layout

├── phase1_compare.py        # first experiment: 3 precisions, 5 prompts
├── evals/
│   ├── eval_set.json        # 70 labeled prompts, 5 categories
│   ├── run_evals.py         # harness: runs ladder, grades, scorecard
│   └── show_failures.py     # failure-analysis helper
├── router/
│   ├── judge.py             # judge-v0 (heuristics) + judge-v1 (LLM)
│   └── evaluate_router.py   # offline policy scoring vs logged outcomes
├── server/
│   └── app.py               # FastAPI adaptive serving layer
└── results/                 # raw eval + routing results (committed)

Stack

Qwen2.5 Instruct 0.5B/1.5B/3B (GGUF, Q4_K_M) · llama.cpp via llama-cpp-python · FastAPI · runs fully local on a MacBook Air, no GPU.

What I'd do next

• Held-out prompt set to test judge generalization
• Confidence-based escalation (retry on the next rung when the small model's answer looks unstable)
• Per-request cost in dollars using cloud price sheets
• Live dashboard visualizing routing decisions and savings