FogGen: Self-Aware Edge–Cloud LLM Router

A 0.6B parameter edge LLM trained to emit a calibrated verbalized confidence score before its answer, enabling efficient edge–cloud routing without an external router.

FogGen is a small, self-aware edge model that knows when to answer locally and when to defer to a stronger cloud model. At inference (figure (a)) it emits a confidence score then an answer in one forward pass; if confidence c ≥ τ the local answer is returned, otherwise the query is routed to the cloud. Training (figure (b)) is a self-evolving loop: each round, the current checkpoint self-samples N=8 generations per question to derive confidence buckets, then SFTs on (question, confidence, answer) triples.

The released checkpoint is the endpoint (R14) of a 14-round chain trained across seven domains: finance, science, coding, law, math, Kazakh culture, medical.

Quick demo

from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("issai/foggen", torch_dtype="bfloat16", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("issai/foggen")

SYSTEM = """You are a self-aware multiple-choice assistant.

Rules:
- Do not output <think> tags.
- First, assess your confidence in solving this question.
- Then give your answer.
- Output format:
  Confidence: <0.0|0.25|0.5|0.75|1.0>
  Final answer: <OPTION_LETTER>"""

question = """A firm reports $400M in total liabilities and $600M in shareholders' equity.
What is the firm's debt-to-equity ratio?

A. 0.67
B. 1.00
C. 1.50
D. 2.00"""

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": question},
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True,
                                       enable_thinking=False).to(model.device)
outputs = model.generate(inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
# Expected:
#   Confidence: 1.0
#   Final answer: A

How routing works

import re

def route_query(model_output: str, tau: float = 0.5):
    """Parse FogGen output. Returns (action, confidence, answer).
    action is 'keep_local' if confidence >= tau, else 'route_to_cloud'."""
    conf_match = re.search(r"Confidence\s*:\s*([\d.]+)", model_output)
    ans_match  = re.search(r"Final\s+answer\s*:\s*([A-D])", model_output)
    if not conf_match: return "route_to_cloud", None, None
    confidence = float(conf_match.group(1))
    answer = ans_match.group(1) if ans_match else None
    return ("keep_local" if confidence >= tau else "route_to_cloud", confidence, answer)

At τ=0.5 on the trained domains, the model routes ~22% of queries to the cloud while achieving 67.8% mean system accuracy.

Model details

Performance

System accuracy at τ=0.5 on seven MCQ domains (full test sets, ~16,200 questions), measured against Random routing and a cloud-only baseline (Qwen3-30B-A3B-Instruct-2507):

Mean lift over Random at τ=0.5: +4.6 (system accuracy minus random-routing accuracy, averaged across the seven domains).

Baseline comparison

Direct comparison against AutoMix (Aggarwal et al., 2024) on the same R14 checkpoint, same evaluation sets:

FogGen achieves higher accuracy at lower cloud cost and 9× lower per-query inference cost.

Open-ended generalization

The MCQ-trained chain transfers to open-ended task types zero-shot. Local accuracy and routing benefit at τ=0.5 on three held-out OE benchmarks:

One additional round of OE training (R15, 1876 SFT rows) lifts local accuracy on these three benchmarks to 86.5% / 40.0% / 58.0% respectively; see issai/foggen-r15-oe.

Citation

Paper coming soon.

Acknowledgements

Thanks to the Qwen team at Alibaba for the base model and cloud teacher.