A Mac-native LLM runtime that's also a coding agent.
One Swift + MLX binary — text, vision, and voice. No Python.

What makes Krill different

Most local-LLM stacks are one half of a pair:

• an engine (Ollama, llama.cpp) that runs a model but can't do anything, and
• a harness (Claude Code, Codex) that drives tools and edits files but borrows someone else's model.

Krill is both, in one binary. The same native Swift + MLX engine that serves tokens also runs a full agent loop — tools, file edits, web fetch, permissions — against the model already sitting in your RAM. Krill is local by default: the native path needs no second process or Python bridge and keeps model traffic on your Mac. Hosted model traffic leaves the machine only when you explicitly select OpenCode or Codex.

One engine, four modes

The agent isn't boxed into your filesystem. web_search ranks the open web — keyless out of the box (DuckDuckGo), or point it at Brave/Tavily (free-tier API key) or your own SearXNG for reliable results — and web_fetch reads any page as clean text — both SSRF-guarded and untrusted-framed against prompt injection — while /research <question> runs a multi-source deep-research pass (plan queries → fetch → summarize each source → synthesize a cited answer). A local model that browses, and cites its work.

And the inverse: krill launch claude (or codex, opencode, copilot, droid, hermes, pi) points an external harness at Krill's engine. Krill is the model for other agents, or the agent on its own model.

Whichever model you point at it, the agent's tool names are constrained while the model samples them — so a model trained on some other harness's vocabulary can't ask for a tool Krill doesn't offer and die on its first call. It's a trigger-activated grammar: idle during prose, armed only inside a tool call, and it costs no measurable decode time (docs/TOOL_NAME_RESOLUTION.md).

Underneath: a continuous batcher (~2× throughput under load), shared-prefix KV reuse (repeat prompts hit cache instead of re-prefilling — the agentic/RAG fast path), speculative decoding, and native vision (SigLIP2) + audio (USM Conformer). All Swift + MLX.

⚠️ Early release. Krill is young and still getting its polish — expect some rough edges, and pin a version if you need stability. Bug reports, ideas, and feedback are genuinely welcome → open an issue.

Install

# Homebrew
brew tap srvsngh99/krill && brew install krill

# …or the one-line installer (Apple Silicon, no Homebrew needed)
curl -fsSL https://raw.githubusercontent.com/srvsngh99/Krill/main/install.sh | sh

The installer verifies the release archive against the SHA-256 digest published by GitHub before extracting it. Set KRILL_VERSION to pin a release.

Updating: Homebrew installs update with brew upgrade krill; installer builds update in place with krill update, which fetches the installer from the target release tag (add --check to only see if a newer release is available).

# Requires macOS 14+ (Apple Silicon, M1+), Swift 6.2+, and the Metal Toolchain
#   xcodebuild -downloadComponent MetalToolchain
git clone https://github.com/srvsngh99/Krill.git && cd Krill
make release && make install      # → /usr/local/bin/krill

Quick start

krill pull gemma-4-e2b                       # text + image + audio, all native

krill run  gemma-4-e2b                        # chat: full-screen TUI
krill run  gemma-4-e2b "explain MLX in one line"   # chat: one-shot
krill code "add a docstring to the top fn in main.swift"   # agent: tools + edits
krill serve --model gemma-4-e2b               # API server on :57455

krill run gemma-4-e2b "what's here?" --image ./photo.png   # vision
krill run gemma-4-e2b "transcribe this"      --audio ./clip.wav   # audio
krill pull unlimited-ocr && krill run unlimited-ocr --image page.png "document parsing."   # OCR

Hosted models in the Krill harness

The local MLX path remains the default. Hosted providers are explicit and use the classic line renderer while Krill continues to own the tool loop and permission checks.

# OpenCode Zen is keyless. The catalog is read live and every `-free` model is
# available automatically; Ox Alpha is preferred while it remains advertised.
krill code --provider opencode --list-models
krill code --provider opencode "explain this repository"
krill code --provider opencode nemotron-3-ultra-free "review this change"

# Codex uses the official CLI's saved ChatGPT login. Krill never reads or
# copies the subscription credential and does not turn it into an API key.
codex login
krill code --provider codex "review this repository"
krill code --provider codex MODEL_ID "review this repository"  # optional model

Omit the Codex model to keep the user's current/default subscription model. Each user signs in with their own codex login; a shared API key is neither required nor supported by this bridge. --base-url applies only to the OpenCode-compatible provider; it is rejected for Codex.

OpenCode defaults to a 4096-token completion budget (use --max-tokens to override it); the local MLX default remains 1024. Codex CLI controls its own completion budget, so --max-tokens does not apply to that provider. A response that reaches the hosted OpenCode budget reports a clear error with the flag to increase it rather than silently treating a partial answer as complete.

Hosted reasoning models spend most of a turn before emitting any content, so OpenCode requests use a 300s ceiling rather than URLSession's 60s default, and transient failures (429/502/503/504, or a dropped connection) are retried twice with a short backoff before the error is surfaced. Free tiers return these routinely; a blip should not end an agent run.

The Codex bridge starts a fresh codex exec --ephemeral process for each agent turn and sends that turn's full transcript to the CLI. It does not provide cross-turn prompt caching, so usage and any applicable charges can grow with long agent runs. Krill invokes the official CLI using its existing login; it does not determine whether a particular account or plan is permitted for a given workflow, so review the terms that apply to your account.

The default port 57455 is unique, so Krill coexists with Ollama on 11434; run krill serve --port 11434 for a literal drop-in.

📖 New to Krill? The User Guide is a single indexed, example-driven walkthrough of every feature — chat & TUI, multimodal (image / audio / OCR), agentic coding, the HTTP server, structured output, embeddings, web search, model management, and configuration.

On Mac, vs Ollama

Single-stream decode is at parity — both hit the MLX memory-bandwidth roof, and Krill makes no raw-decode-speed claim. Krill leads where real workloads live:

• Capability — native vision + audio + voice, grammar-constrained output, schema / tool_choice tool calls. Ollama's MLX Gemma tag has none.
• Concurrency — ~2× aggregate throughput under load (continuous batcher vs serialized).
• Latency — faster cold start; shared-prefix KV turns repeat-prefix / agentic / RAG calls into cache hits. Gemma-4-E2B: TTFT ~5×, wall ~1.57× faster.
• Memory — ~3 GB vs ~8.8 GB peak (class-equal gate; capped Metal buffer pool).

Numbers track your hardware — reproduce them, don't trust a banner:

make bench-compare KRILL_MODEL=llama-3.2-1b OLLAMA_MODEL=llama3.2:1b

Full methodology and gates: docs/BENCHMARKS.md, docs/BENCHMARKING.md.

Models

37 chat & multimodal models ship as one-word krill pull shortcuts (plus ~19 embedding / reranker models), spanning ~15 architecture families. Switch between installed models live in a chat with /model — the conversation carries over — or import any mlx-community repo and it joins the picker.

krill pull gemma-4-e2b       # Gemma 4 — text + image + audio, all native (also: -e4b, -12b flagship)
krill pull qwen3.8-27b       # Qwen 3.8 27B — hybrid linear-attention VLM, text + image (~16 GiB)
krill pull qwen3-14b         # Qwen 3 (incl. MoE: qwen3-30b) — also Qwen 2.5: qwen2.5-7b
krill pull llama-3.2-3b      # Llama 3.2 / 3.1 (also: llama-3.2-1b, llama-3.1-8b)
krill pull mistral-7b        # Mistral 7B v0.3
krill pull gemma-2-9b        # Gemma 2 9B
krill pull phi-4-mini        # Phi-4 Mini
krill pull unlimited-ocr     # native document/image OCR (DeepSeek-OCR)

krill pull mlx-community/Meta-Llama-3.1-8B-Instruct-4bit   # …or any mlx-community repo

Native text also runs Phi, GLM-4, Mixtral, OLMoE, and DeepSeek-V2/V3, plus a ~15-family embedding/reranker stack. Vision serving adds LLaVA-1.5, Llama-3.2-Vision (mllama, multi-image), and Qwen2.5-VL.

Formats: not anything-goes — Krill is MLX-native. It runs MLX-format checkpoints (safetensors) in 4-bit, 8-bit, nvfp4 (mixed-precision 4-bit-float), or bf16/fp16 — GGUF is not supported. Any mlx-community model of a supported architecture loads as-is; convert other Hugging Face checkpoints with krill quantize <hf-path>.

Commands

Faster CLI: krill run reloads the model each call. Start a daemon once and subsequent calls auto-route to it — TTFT drops from seconds to milliseconds:

KRILL_KEEP_ALIVE=24h krill serve --model qwen2.5-3b &
krill run qwen2.5-3b "hi"     # auto-routed → "(via daemon @ :57455)"

API compatibility

One server speaks three protocols — change the port in your client, nothing else:

• OpenAI — /v1/chat/completions (SSE), /v1/completions, /v1/responses, /v1/models
• Ollama — /api/chat, /api/generate, /api/tags
• Anthropic — /v1/messages (Claude SDK drop-in)

from openai import OpenAI
client = OpenAI(base_url="http://localhost:57455/v1", api_key="not-used")
print(client.chat.completions.create(
    model="llama-3.2-1b", messages=[{"role": "user", "content": "hi"}]
).choices[0].message.content)

OpenAI-family SDKs (openai, langchain-openai, llama-index) use …/v1; the Anthropic SDK takes the bare host (it appends its own /v1/messages). Request shapes and more SDKs: docs/SERVER_API.md, docs/CONNECT_CODING_AGENTS.md.

The agent on your phone

krill serve also serves a full agent UI at /ui — no app store, no extra install, it ships inside the binary. Point the agent at any repo on your Mac from your phone's browser: sessions, live tool-call transcript, and Deny / Allow / Always approval cards for every mutating tool.

krill ui

That one command starts the server in the background (it keeps running after you close the terminal), generates an API key on first run and saves it to ~/.krill/config.toml, prints the links, and opens the UI on the Mac:

>_ Krill UI
This Mac    http://localhost:57455/ui
Same Wi-Fi  http://192.168.1.23:57455/ui
Tailscale   http://100.91.59.7:57455/ui
Phone link  http://100.91.59.7:57455/ui#k=…   (carries the key)
Stop        krill ui --stop      always-on at login: krill ui --install

1. Open the phone link on the phone (same Wi-Fi, or anywhere with Tailscale on both) — the key is filled in for you.
2. iPhone: Share → Add to Home Screen → a standalone app with the Krill tile.
3. Leave it on: krill ui --install registers the server as a login item — it starts when you log in and restarts if it ever exits. krill ui --uninstall removes it; krill ui --status reprints the links.

Prefer to run it yourself? KRILL_API_KEY='…' krill serve --host 0.0.0.0 is the same server (a non-loopback bind requires a key). Never port-forward it raw to the internet — the agent runs tools on your Mac; keep it behind a VPN or an authenticated tunnel. Full setup, endpoint contract, and development notes: docs/AGENT_UI.md.

Architecture

One Swift package, no Python. The engine and the agent harness share a process and the same loaded model.

Engine

• KrillEngine — orchestration, continuous batcher, speculative decoding
• KrillCore — architectures (Llama, Qwen/MoE, Mistral, Gemma 4, Phi, GLM, DeepSeek …) + SigLIP2 vision + USM Conformer audio
• KrillCache — KV cache (fp16 / int8, prefix reuse) · KrillKernels — fused Metal shaders · KrillSampler · KrillGrammar — grammar-constrained decode

Harness

• KrillHarness — agent loop, permissions, and tools (bash, read/write/edit, glob/grep, web fetch/search, dispatch, deep research)
• KrillAgent — hardware-aware operator / recommender

Surfaces

• KrillServer — OpenAI/Ollama/Anthropic HTTP (swift-nio) · KrillCLI + KrillTUI — CLI and the full-screen chat/agent TUI · KrillRegistry — model store + HF puller · KrillTokenizer

The full-screen TUI (themes, slash commands, attachments, push-to-talk voice) has its own reference: docs/TUI.md.

Configuration

~/.krill/config.toml, or environment variables:

Web search

web_search works with no setup via DuckDuckGo. For reliable, rate-limit-free results, add a free-tier API key:

krill --config search_backend=brave         # or: tavily
krill --config brave_api_key=YOUR_KEY        # or export KRILL_BRAVE_API_KEY

Or point at a self-hosted SearXNG with search_backend=searxng + searxng_url. See docs/decisions/0002-web-search-backends.md.

Author

Sourav Singh / Sourav AI Labs · souravailabs.ai

License

MIT