ExLlamaSharp

Local LLM server for Windows + NVIDIA — EXL3 inference, OpenAI-compatible /v1 API, and a full Blazor admin UI. No Docker. No WSL required.

 

 

Built for small teams and office GPU boxes (roughly 5–50 people) that want an Ollama-like workflow with API keys, audit, jobs, and multi-GPU — as a native Windows service.

Inspired by Ollama, vLLM, ExLlamaV3, and Open WebUI-style admin.

---

Download (Windows x64)

↓ ExLlamaSharp-Setup-win-x64.exe — latest GitHub Release

irm https://raw.githubusercontent.com/Kortexio/ExLlamaSharp/main/packaging/install-web.ps1 | iex

Then open http://127.0.0.1:14563

---

Why ExLlamaSharp

Full matrix: docs/comparison.md

---

Quick start

1. Install with Setup.exe (or the one-liner above).
2. Sign in to the Admin UI → Models → Library (search exl3) → Download.
3. My Models → Load (pick a VRAM profile that fits).
4. Chat in the UI, or point any OpenAI client at http://127.0.0.1:14563/v1 with your API key.

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:14563/v1", api_key="sk-exllamasharp-dev")
print(client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Hello from ExLlamaSharp"}],
))

More: docs/quick-install.md · docs/README.md

---

Current release

1.4.1 — Tray/Admin update checks (GitHub Releases → About download), English load-profile labels, English installer scripts. Builds on 1.4.0: Multi-GPU (pipeline / tensor), combined VRAM fit, CUDA remap, AdminLTE UI, EXL3 OpenAI chat, Models/Jobs/Keys, LoRA, speculative decoding, webhooks, tenants. Setup.exe bundles ExLlamaV3 CUDA .pyd, worker deps, Python installer and VC++.

---

Supported models

ExLlamaSharp runs EXL3 models only (the ExLlamaV3 quantized format) on an NVIDIA GPU.

A model is a folder that looks like a Hugging Face snapshot:

How to get one:

1. Admin UI → Models → Library (default search exl3) → Download
2. Or import a local folder that already has those files
3. My Models → Load

Hugging Face repos often keep the actual weights on a bitrate branch such as 4.00bpw / 4.0bpw, not on main. The server resolves that revision automatically and will not treat a README-only clone as a successful download.

Examples that work:

• turboderp/MiniCPM5-1B-exl3
• turboderp/Llama-3.2-1B-Instruct-exl3 (revision 4.0bpw) — small demo / first-run smoke test

Search Hugging Face for exl3 (many IDs end in -exl3). Admin → Models shows a Fits / Tight / Too large badge against this machine’s GPU VRAM (estimate from weight size; it does not auto-select a model).

Not supported (will not load for real inference):

LoRA adapters can be registered in the UI/API and applied at generation time via the EXL3 worker (X-Adapter-Id or worker load_adapter).

Chat uses the tokenizer’s Hugging Face chat template when present (Llama 3 / ChatML fallbacks otherwise).

---

Features

This section describes every product surface in the current build: installer, tray, Windows service, Admin UI pages, and APIs. Where something is still a stub, that is called out.

Windows installer and service

The Setup.exe (Inno Setup) installs a self-contained .NET host. End users do not need the .NET SDK, CUDA Toolkit, or LibTorch.

Firewall rule and Start Menu icon are created by the installer. The service listens on 14563 so it does not collide with common local ports (8080, 8787).

System tray

ExLlamaSharp.Tray.exe lives in the notification area (single instance). It:

• Polls service + http://127.0.0.1:14563/health every few seconds
• Shows green (healthy), yellow (service up but health failed), or grey (stopped)
• Open Admin UI (double-click or menu)
• Open data folder (%ProgramData%\ExLlamaSharp)
• Start / Stop / Restart the Windows service
• Registers itself in HKCU Run so it starts with the user session

Inference backends

Python resolution order: EXLLAMASHARP_PYTHON, exl3-runtime.json, the app venv, or a repo .venv-exl3. Set EXL3_BC_DSA=0 for current ExLlamaV3 workers.

Chat templates: the worker prefers tokenizer.apply_chat_template. If that is missing, Llama 3 special tokens are used when present; otherwise ChatML (<|im_start|> / <|im_end|>). Special tokens are stripped from streamed replies.

OpenAI-compatible API (/v1)

Use any OpenAI SDK. Point base_url at http://127.0.0.1:14563/v1 and send Authorization: Bearer <key>.

Auth: API keys with scopes (chat, completions, embeddings, admin). Per-key RPM/TPM limits return 429.

Admin API (/api/v1)

Ops (no API key): GET /health, GET /ready, GET /metrics (Prometheus).

---

Admin UI (Blazor)

Design system: Kortexio theme (DM Sans / Fraunces, teal accent). Login cookie + the built-in Admin API key are used so Chat and library calls work from the browser.

Workspace

Dashboard (/)
Server overview: process status, requests today, loaded model name, GPU utilization / name. Optional advanced cards: tokens/sec, jobs waiting/running, VRAM. Toggle “Show advanced metrics”. First-run onboarding state is stored in %ProgramData%\ExLlamaSharp\ui-state.json.

Chat (/chat)
Playground that streams POST /v1/chat/completions. Shows whether a real model is loaded or the mock engine is answering. Enter sends; Shift+Enter is not required (single-line input). If nothing is loaded, the page tells you to open Models → My Models → Load.

Models (/models)
Three tabs:

• Library — live Hugging Face search (default query exl3). Shows name, repo id, parameter label, size, and Download. Gated repos need a token in Settings → Hugging Face. For EXL3 repos that keep weights on branches such as 4.00bpw (not main), the server resolves the revision automatically and refuses a “success” that only downloaded a README.
• My Models — folders under the models path that contain config.json (scanned from disk). Load puts that model on the GPU.
• Import — register an existing local folder + alias (does not copy files).

Jobs (/jobs)
Queue for pull / quantize / import. Cards for active, waiting, and recent (completed / failed / cancelled). Polls every 2 seconds. Pull jobs show downloaded/total bytes and parameter label. Cancel is available while a job is pending or running. Use Refresh if the Blazor poll looks stuck.

API Keys (/keys)
Create named keys with scopes. The plaintext secret is shown once. List and revoke existing keys. Use these from apps, curl, or the OpenAI SDK.

Usage (/usage)
Business view of the audit trail: requests in the last 7 days, prompt/completion tokens, estimated cost, and a recent activity table (endpoint, tokens, status, latency).

Team (/team)
Lists users who can manage the server (username, role, tenant, last active). Create the first admin in Setup if the list is empty.

Advanced (sidebar toggle)

Adapters (/adapters)
Register/delete LoRA adapters (name, path). Apply at inference with header X-Adapter-Id on chat completions (worker loads one PEFT LoRA globally on the model).

Metrics (/dashboard/metrics)
Live tokens/sec and job counts. A/B tab creates tests and records votes; traffic is tagged on audit.

API Guide (/api)
Quick reference for /v1 and /api/v1 endpoints.

Logs (/logs)
Live in-memory tail (start/stop), min level, text filter, plus refresh of persisted audit rows.

Diagnostics (/diagnostics)
Runs /health and /ready. Component cards (database, engine, inference, disk, …) and a short list of common fixes (no model loaded, missing nvidia-smi, port in use, API 401).

Tenants (/admin/tenants)
Create/list tenants in SQLite (id, name, subdomain). HTTP /api/v1/tenants is wired to the same DB; isolation at the request layer is not complete.

System

Settings (/settings)
Persisted server settings:

Setup (/setup)
Five-step wizard: welcome + GPU detect → create admin → models path → network (localhost vs LAN) → finish / optional starter model. Re-runnable from the sidebar.

API Guide (/api)
Copy-paste curl examples against the live base URL (/v1/models, chat completions, …).

About (/about)
Version, build date, .NET/OS, engine (mock/loaded/path/TPS), GPU name and VRAM.

Login (/login)
Username/password for the Admin UI. Default seed on a fresh database: admin / changeme.

---

Platform and ops

• Windows Service + tray autostart
• API key auth and per-key rate limits
• Async audit trail (SQLite)
• Scheduled / on-demand backup
• Live log tail (UI + SSE)
• SignalR dashboard hub (subscribe / ping)
• PWA manifest / service worker on the Admin UI
• Self-contained publish (no .NET SDK on the target PC)

Still stub / partial

By design (not product gaps for the EXL3 text + vision chat product):

• OpenAI images / audio generation — 501; separate Media track
• Native DLL generate — worker-only for production text; DLL remains CI / scheduler ABI
• MCP / hosted ReAct agent — not embedded; use OpenAI tools / tool_calls with your own agent loop
• Multi-GPU model-parallel (MP) — not supported. Tensor and layer autosplit (pipeline) are supported via the EXL3 worker on N NVIDIA GPUs (VRAM-proportional split, optional GpuSplitGb).

A/B tests: create via /api/v1/ab, then send X-Ab-Test-Id (or model: "ab:<guid>") on chat/completions. The server assigns A/B via consistent hash, may load the selected model when it differs from the one currently on the GPU, tags audit, and returns X-Ab-Variant.

Embeddings: require ONNX under %ProgramData%\ExLlamaSharp\embeddings\all-MiniLM-L6-v2\model.onnx (dim 384). Without it, /v1/embeddings returns 503 unless EXLLAMASHARP_ALLOW_EMBEDDING_FALLBACK=1 (CI only).

Tools: request tools → response may include message.tool_calls and finish_reason: "tool_calls". Clients run the tool and send role tool turns.

Vision: with an EXL3 VLM loaded (vision component present — e.g. Qwen3-VL, Gemma VL), image_url / data URLs are encoded via get_image_embeddings and injected into the Job. Text-only models return 400 vision_not_supported.

Quick start

End users

1. Install from the latest Release (or the one-liner above).
2. Open http://127.0.0.1:14563 and sign in (admin / changeme).
3. Models → Library — search exl3 (for example turboderp/MiniCPM5-1B-exl3) and Download. Watch Jobs until completed.
4. Models → My Models → Load.
5. Open Chat and send a message.

If inference fails after a custom Python repair, run Setup-Exl3Python.bat in the install folder. The Setup.exe already installs the official CUDA wheel — do not pip install exllamav3 from PyPI.

Developers (run from source)

dotnet restore ExLlamaSharp.slnx
dotnet build ExLlamaSharp.slnx -c Release
dotnet run --project src/ExLlamaSharp.Server/ExLlamaSharp.Server.csproj

Open http://127.0.0.1:14563.

Build the installer

powershell -ExecutionPolicy Bypass -File packaging\Build-Installer.ps1
# → publish\ExLlamaSharp-Setup-win-x64.exe
# → publish\ExLlamaSharp-Setup-win-x64.zip

Details: docs/INSTALL.md · packaging/README.md.

---

Requirements

Data directory: %ProgramData%\ExLlamaSharp\ (app.db, models\, logs\, backups\).

Override for tests: environment variable EXLLAMASHARP_DATA_ROOT.

---

Architecture

┌─────────────────────────────────────────┐
│  Browser PWA (any PC on the LAN)        │
│  Blazor UI (Kortexio theme)             │
└─────────────────────────────────────────┘
              ↓ HTTP
┌─────────────────────────────────────────┐
│  Windows Service / console (GPU host)   │
│  ┌───────────────────────────────────┐  │
│  │ Kestrel .NET 10                   │  │
│  │ ├─ OpenAI /v1                     │  │
│  │ ├─ Admin /api/v1                  │  │
│  │ └─ Blazor Server + PWA            │  │
│  └───────────────────────────────────┘  │
│  ┌───────────────────────────────────┐  │
│  │ ExLlamaV3WorkerEngine (Python)    │  │
│  │ tools/exl3_worker → ExLlamaV3     │  │
│  │ Real EXL3 CUDA kernels            │  │
│  └───────────────────────────────────┘  │
│              ↓ optional P/Invoke         │
│  ┌───────────────────────────────────┐  │
│  │ exllamasharp_native.dll (C++/CUDA)│  │
│  │ Scheduler, PageTable, EXL3 check  │  │
│  └───────────────────────────────────┘  │
│  SQLite · audit · backup · live logs    │
└─────────────────────────────────────────┘

More detail: docs/architecture.md.

Important: Do not push changes to upstream turboderp-org/exllamav3. Keep a local third_party/exllamav3 tree only (not in this git repo).

---

Documentation

Index: docs/README.md

• User Manual · Admin Guide · API Reference
• Troubleshooting · Architecture · Comparison
• Packaging · Swagger (Development): /swagger
• Contributing: CONTRIBUTING.md · Security: SECURITY.md

---

Development

# Prerequisites (as needed)
# - .NET 10 SDK
# - Visual Studio 2022+ with C++ (native builds)
# - CUDA Toolkit 12.8+ (native CUDA DLL)
# - CMake 3.25+
# - Python 3.11+ (real EXL3 worker)

dotnet restore ExLlamaSharp.slnx
dotnet build ExLlamaSharp.slnx -c Release

# Native stub (CI / no GPU toolkit)
.\packaging\build-native-stub.ps1

# Native CUDA (optional; validates EXL3 dirs)
.\packaging\build-native-cuda.ps1

# Real EXL3 path
.\packaging\Setup-Exl3Python.ps1
.\packaging\Download-DemoModel.ps1

dotnet run --project src/ExLlamaSharp.Server/ExLlamaSharp.Server.csproj
dotnet test ExLlamaSharp.slnx

# E2E feature matrix
dotnet test --filter FullyQualifiedName~E2eFeatureMatrix -c Release

Publish + Windows install:

.\packaging\Build-Installer.ps1

See third_party/README.md and packaging/install-cuda-libtorch.md.

---

License

Apache 2.0

Acknowledgments

• ExLlamaV3 — EXL3 CUDA kernels and model format
• vLLM — serving / scheduler inspiration
• Ollama — UX inspiration