33
stars
37
commits
2
linked in READMEs
Sep 10, 2026
updated
README
Edge0-8b-a1b Preview
An 8B-class sparse MoE that runs in phone-class memory.
1 GiB active memory · 25 tok/s · 4-bit
Edge0-8b-a1b — an 8B MoE LLM that runs at viable speed in under 1 GiB of active memory, via the edge0 streaming inference framework.
Preview status: this is an early preview release of the edge0 pipeline. The checkpoint ships as int4 quantization plus LoRA and prerouter adapters trained for this framework.
Highlights
- Runs in phone-class memory: the full 4-bit checkpoint stays on storage and experts are streamed on demand, so only the active weights are in RAM — under 1 GiB, with no sharding and no upfront download of the weights into memory.
- Fast enough for interactive use: 25 tok/s decode; long prompts fill in at 1400 tok/s.
- Quality kept after quantization: Recover-LoRA distillation keeps the int4 model within 2.8 points of its fp16 base (and above it on MMLU-Pro).
- Works out of the box: base, LoRA and prerouter adapters ship
together and load automatically via
edge0.
Three mechanisms make this work:
- SSD expert offload: expert weights are streamed from storage on demand — fetched only as routed, so RAM holds just the active weights. Peak memory is bounded by the active set, not the parameter count.
- Prerouter: a trained head predicts expert routing one step ahead, so expert loads overlap the forward pass instead of stalling it — up to +59% decode throughput; the gain grows with storage latency, model size, and routed width K.
- Recover-LoRA: the int4 base is frozen and LoRA adapters are trained by distillation from the FP teacher, recovering most of the quantization loss at 4-bit (see Quality below). Adapters stay unmerged: one read-only base serves multiple adapter sets.
Model summary
| Base model | inclusionAI Ling 3.0 tiny (bailing hybrid, MLA + MoE, ≈7.9B total / ≈1.2B active) |
| Quantization | 4-bit |
| Layers | 24 |
| Experts / active per token | 128 / 8 (K=8) |
| Hidden size | 1536 |
| Context | 128k |
| Thinking mode | yes (chat template) |
| License | Apache 2.0 |
| Framework | edge0 (MLX backend) |
| Contents | base checkpoint + lora_edge0_8b.safetensors + prerouter_edge0_8b.safetensors |
The LoRA and prerouter adapters are co-located with the base checkpoint
and load automatically — this repository is a complete, ready-to-run
model directory for edge0.
Quality
All benchmarks were run by us with OpenCompass under identical settings and parameters for both models. The loss of the edge0 pipeline (int4 + adapters) relative to the fp16 base model is small: 2.8 points on average, with MMLU-Pro above the base. Max 100:
| Benchmark | edge0-8b (int4) | Ling 3.0 tiny (fp16) |
|---|---|---|
| AIME 2026 | 63.3 | 73.3 |
| HumanEval | 91.5 | 92.7 |
| GPQA-Diamond | 70.7 | 71.2 |
| MMLU-Pro | 70.1 | 65.8 |
| IFBench | 53.9 | 60.6 |
| Average | 69.9 | 72.7 |
Performance
Measured with examples/bench.py on a Mac mini M4 Pro, 24 GB:
| Decode speed | Prefill throughput (cold / warm) | Peak active memory* |
|---|---|---|
| 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB |
*Short contexts; long contexts add KV cache (≈3.3 GiB at 3.3k tokens). Expert weights stream from SSD on demand and are not resident.
Use cases
- Edge / on-device inference where GPU VRAM is scarce and storage is fast (NVMe, internal flash).
- Batch serving on a single commodity machine — one read-only base serves many LoRA adapter sets without re-quantization.
- Multilingual chat and reasoning with thinking mode enabled by the bundled chat template.
Limitations
- Preview release: coverage and quality are still being extended; the model is primarily tuned for the languages of the base model.
- Agent capability: this preview release is not yet optimized for agentic tasks — tool use, multi-step planning, and long-horizon autonomy are currently weak. The full release will substantially strengthen agent capability.
- The MLX backend currently targets Apple Silicon; other backends are on the edge0 roadmap.
- Long contexts grow the KV cache (≈3.3 GiB at 3.3k tokens); use shorter contexts to keep peak memory at 1 GiB.
Quick start
pip install -e 'git+https://github.com/Edge0-AI/edge0.git#egg=edge0[fetch]'
# Download this repository into a local directory
huggingface-cli download Edge0/Edge0-8b-a1b-preview --local-dir ./Edge0-8b-a1b-preview
# Run it
export EDGE0_8B_MODEL=$PWD/Edge0-8b-a1b-preview
edge0 chat --name edge0-8b --prompt "Introduce yourself"
# Or serve an OpenAI-compatible HTTP API
edge0 serve --name edge0-8b --port 8083
For full usage (Python API, streaming options, prerouter details), see the edge0 documentation.
License
Apache 2.0. See LICENSE.
Similar repos
33
stars
37
commits
2
linked in READMEs
Sep 10, 2026
updated
README
Edge0-8b-a1b Preview
An 8B-class sparse MoE that runs in phone-class memory.
1 GiB active memory · 25 tok/s · 4-bit
Edge0-8b-a1b — an 8B MoE LLM that runs at viable speed in under 1 GiB of active memory, via the edge0 streaming inference framework.
Preview status: this is an early preview release of the edge0 pipeline. The checkpoint ships as int4 quantization plus LoRA and prerouter adapters trained for this framework.
Highlights
- Runs in phone-class memory: the full 4-bit checkpoint stays on storage and experts are streamed on demand, so only the active weights are in RAM — under 1 GiB, with no sharding and no upfront download of the weights into memory.
- Fast enough for interactive use: 25 tok/s decode; long prompts fill in at 1400 tok/s.
- Quality kept after quantization: Recover-LoRA distillation keeps the int4 model within 2.8 points of its fp16 base (and above it on MMLU-Pro).
- Works out of the box: base, LoRA and prerouter adapters ship
together and load automatically via
edge0.
Three mechanisms make this work:
- SSD expert offload: expert weights are streamed from storage on demand — fetched only as routed, so RAM holds just the active weights. Peak memory is bounded by the active set, not the parameter count.
- Prerouter: a trained head predicts expert routing one step ahead, so expert loads overlap the forward pass instead of stalling it — up to +59% decode throughput; the gain grows with storage latency, model size, and routed width K.
- Recover-LoRA: the int4 base is frozen and LoRA adapters are trained by distillation from the FP teacher, recovering most of the quantization loss at 4-bit (see Quality below). Adapters stay unmerged: one read-only base serves multiple adapter sets.
Model summary
| Base model | inclusionAI Ling 3.0 tiny (bailing hybrid, MLA + MoE, ≈7.9B total / ≈1.2B active) |
| Quantization | 4-bit |
| Layers | 24 |
| Experts / active per token | 128 / 8 (K=8) |
| Hidden size | 1536 |
| Context | 128k |
| Thinking mode | yes (chat template) |
| License | Apache 2.0 |
| Framework | edge0 (MLX backend) |
| Contents | base checkpoint + lora_edge0_8b.safetensors + prerouter_edge0_8b.safetensors |
The LoRA and prerouter adapters are co-located with the base checkpoint
and load automatically — this repository is a complete, ready-to-run
model directory for edge0.
Quality
All benchmarks were run by us with OpenCompass under identical settings and parameters for both models. The loss of the edge0 pipeline (int4 + adapters) relative to the fp16 base model is small: 2.8 points on average, with MMLU-Pro above the base. Max 100:
| Benchmark | edge0-8b (int4) | Ling 3.0 tiny (fp16) |
|---|---|---|
| AIME 2026 | 63.3 | 73.3 |
| HumanEval | 91.5 | 92.7 |
| GPQA-Diamond | 70.7 | 71.2 |
| MMLU-Pro | 70.1 | 65.8 |
| IFBench | 53.9 | 60.6 |
| Average | 69.9 | 72.7 |
Performance
Measured with examples/bench.py on a Mac mini M4 Pro, 24 GB:
| Decode speed | Prefill throughput (cold / warm) | Peak active memory* |
|---|---|---|
| 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB |
*Short contexts; long contexts add KV cache (≈3.3 GiB at 3.3k tokens). Expert weights stream from SSD on demand and are not resident.
Use cases
- Edge / on-device inference where GPU VRAM is scarce and storage is fast (NVMe, internal flash).
- Batch serving on a single commodity machine — one read-only base serves many LoRA adapter sets without re-quantization.
- Multilingual chat and reasoning with thinking mode enabled by the bundled chat template.
Limitations
- Preview release: coverage and quality are still being extended; the model is primarily tuned for the languages of the base model.
- Agent capability: this preview release is not yet optimized for agentic tasks — tool use, multi-step planning, and long-horizon autonomy are currently weak. The full release will substantially strengthen agent capability.
- The MLX backend currently targets Apple Silicon; other backends are on the edge0 roadmap.
- Long contexts grow the KV cache (≈3.3 GiB at 3.3k tokens); use shorter contexts to keep peak memory at 1 GiB.
Quick start
pip install -e 'git+https://github.com/Edge0-AI/edge0.git#egg=edge0[fetch]'
# Download this repository into a local directory
huggingface-cli download Edge0/Edge0-8b-a1b-preview --local-dir ./Edge0-8b-a1b-preview
# Run it
export EDGE0_8B_MODEL=$PWD/Edge0-8b-a1b-preview
edge0 chat --name edge0-8b --prompt "Introduce yourself"
# Or serve an OpenAI-compatible HTTP API
edge0 serve --name edge0-8b --port 8083
For full usage (Python API, streaming options, prerouter details), see the edge0 documentation.
License
Apache 2.0. See LICENSE.