wandler

OpenAI-compatible inference server powered by transformers.js. Run ONNX models locally with WebGPU acceleration or CPU — no Python, no CUDA required.

Think vLLM or llama.cpp, but for the ts crowd.

Quickstart

npx wandler --llm onnx-community/gemma-4-E4B-it-ONNX:q4

# custom model, precision, device, port
npx wandler --llm LiquidAI/LFM2.5-1.2B-Instruct-ONNX:fp16 --device cpu --port 3000

# with embeddings and STT
npx wandler --llm onnx-community/gemma-4-E4B-it-ONNX:q4 \
  --embedding Xenova/all-MiniLM-L6-v2:q8 \
  --stt onnx-community/whisper-tiny:q4

Use it with the OpenAI SDK:

import OpenAI from "openai";

const client = new OpenAI({ baseURL: "http://localhost:8000/v1", apiKey: "-" });

const response = await client.chat.completions.create({
  model: "onnx-community/gemma-4-E4B-it-ONNX",
  messages: [{ role: "user", content: "Hello!" }],
  stream: true,
});

for await (const chunk of response) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

CLI

wandler — transformers.js inference server

Usage:
  wandler --llm org/repo[:precision] [options]
  wandler model ls [--type <type>]

Commands:
  models                    List available models from the catalog

Model:
  -l, --llm <id>              LLM model
      --backend <name>        LLM backend: wandler, transformersjs (default: wandler)
  -e, --embedding <id>        Embedding model
  -s, --stt <id>              STT model
  -d, --device <type>         Device: auto, cpu, cuda, coreml, dml, webgpu, wasm (default: auto)
      --hf-token <token>      HuggingFace token for gated models
      --cache-dir <path>      Model cache directory

Server:
  -p, --port <number>         Port (default: 8000)
      --host <addr>           Bind address (default: 127.0.0.1)
  -k, --api-key <key>         API key for auth (or WANDLER_API_KEY)
      --cors-origin <origin>  Allowed CORS origin (default: *)
      --max-tokens <n>        Max tokens per request (default: 2048)
      --max-concurrent <n>    Max concurrent requests (default: 1)
      --timeout <ms>          Request timeout in ms (default: 120000)
      --log-level <level>     debug, info, warn, error (default: info)
      --quiet                 Suppress non-error startup/profile logs
      --prefill-chunk-size <n>
                              Chunk size for long-prompt prefill; auto uses a 640MB GPU attention budget; auto:<mb> customizes it; 0/off disables it
      --decode-loop <mode>     Wandler decode loop: auto/on/off (default: auto; on is experimental)
      --prefix-cache <mode>   Enable prefix KV cache: true/false (default: true)
      --prefix-cache-entries <n>
                              Prefix KV cache entries (default: 2)
      --prefix-cache-min-tokens <n>
                              Minimum prefix tokens to cache (default: 512)
      --warmup-tokens <n>     Approximate prompt tokens to run once before serving
      --warmup-max-new-tokens <n>
                              Max new tokens for startup warmup

Info:
  -v, --version               Show version
  -h, --help                  Show this help

Precision suffixes: q4, q8, fp16, fp32 (default: q4)

Environment Variables

Every CLI flag has a corresponding environment variable:

Endpoints

LLM

Embeddings

Audio

Utilities

Parameters

Chat & Text Completions

Standard OpenAI parameters:

Extended parameters (vLLM/llama.cpp compatible):

Embeddings

Compatible Models

List all verified models with their capabilities:

wandler model ls

type      | size  | prec | capabilities             | repo:precision                                   | name
------------------------------------------------------------------------------------------------------------------------
llm       | 2B    | q4   | chat, tool-calling       | onnx-community/gemma-4-E4B-it-ONNX:q4            | Gemma 4 E4B
llm       | 1.2B  | q4   | chat, tool-calling       | LiquidAI/LFM2.5-1.2B-Instruct-ONNX:q4            | LFM 2.5 1.2B
llm       | 350M  | q4   | chat, tool-calling       | LiquidAI/LFM2.5-350M-ONNX:q4                     | LFM 2.5 350M
llm       | 0.8B  | q4   | chat, tool-calling       | onnx-community/Qwen3.5-0.8B-Text-ONNX:q4         | Qwen 3.5 0.8B
llm       | 1.7B  | q4   | chat                     | HuggingFaceTB/SmolLM2-1.7B-Instruct:q4           | SmolLM2 1.7B
embedding | 22M   | q8   | embedding                | Xenova/all-MiniLM-L6-v2:q8                       | all-MiniLM-L6-v2
embedding | 33M   | q8   | embedding                | Xenova/bge-small-en-v1.5:q8                      | BGE Small EN v1.5
embedding | 137M  | q8   | embedding                | nomic-ai/nomic-embed-text-v1.5:q8                | Nomic Embed Text v1.5
stt       | 39M   | q4   | transcription            | onnx-community/whisper-tiny:q4                   | Whisper Tiny
stt       | 74M   | q4   | transcription            | onnx-community/whisper-base:q4                   | Whisper Base
stt       | 244M  | q4   | transcription            | onnx-community/whisper-small:q4                  | Whisper Small

Filter by type:

wandler model ls --type llm
wandler model ls --type embedding
wandler model ls --type stt

Use the repo:precision value directly with --llm, --embedding, or --stt.

Any ONNX model from onnx-community or transformers.js compatible models should work beyond the verified catalog.

Tool Calling

wandler parses tool calls from multiple model output formats:

• LFM: [func_name(arg="val")] and [tool_calls [{...}]]
• Qwen: <tool_call>{"name": "...", "arguments": {...}}</tool_call>
• OpenAI JSON: {"tool_calls": [...]}

Thinking blocks (<think>...</think>) are automatically stripped before parsing.

License

MIT