InferHub

A self-hosted, Ollama-compatible inference mesh. Run the gateway where you have no GPU. Run worker nodes where you do.

---

The problem

GPUs and servers rarely live in the same place. The machine that is always on — a small VPS, a home server — usually has no GPU. The machines that do have a GPU — a desktop, a gaming rig, a workstation — are often behind a home router and not reachable from outside.

InferHub closes that gap. A lightweight coordinator runs on the always-on, GPU-less host and speaks a familiar Ollama-style API. The actual work runs on nodes that sit on your GPU machines, reach out to the coordinator, and pull jobs down. No port forwarding, no exposing your desktop to the internet.

In short: one stable address in front, a pool of GPUs behind it.

How it works

                                  ┌──────────────────────────────┐
   client (Ollama-compatible)     │          Coordinator         │
   curl / app / IDE plugin  ─────▶│  Ollama-style HTTP API        │
        (Bearer token if remote)  │  /api/tags /api/generate      │
                                  │  /api/chat                    │
                                  │                               │
                                  │  SignalR node hub  ◀──────────┼──── persistent outbound
                                  └───────────────┬───────────────┘     connection from nodes
                                                  │
                         dispatch (model-aware)   │   stream tokens back
                          ┌───────────────────────┼───────────────────────┐
                          ▼                       ▼                       ▼
                   ┌────────────┐          ┌────────────┐          ┌────────────┐
                   │   Node A   │          │   Node B   │          │   Node C   │
                   │  + Ollama  │          │  + Ollama  │          │  + Ollama  │
                   │  (GPU)     │          │  (GPU)     │          │  (GPU)     │
                   └────────────┘          └────────────┘          └────────────┘

• Coordinator — an ASP.NET Core service. It exposes an Ollama-compatible HTTP API, authenticates remote callers with a Bearer token, keeps track of which nodes are online and which models each one holds, and routes every request to a capable node.
• Node — a small .NET worker. It opens a persistent outbound connection to the coordinator (so it works fine behind NAT), reports which models it can serve, then receives prompts, runs them against its local inference backend, and streams the result back.
• Pluggable backends — each node runs an LLM backend behind a small abstraction (IInferenceBackend). The first backend is Ollama, driven by our own OllamaClient library. The mesh isn't tied to it — other backends (vLLM, llama.cpp, OpenAI-compatible servers) can slot in later without touching the coordinator.

Because the API is Ollama-shaped, existing Ollama clients, scripts, and editor plugins can point at the coordinator and keep working — they just reach a whole pool of GPUs instead of one.

Status

InferHub 3.50 gives a desktop its card back. Every release before it assumed the GPU belonged to the node — Ollama resident for its own keep_alive, every tool holding a warm worker — which is right on a dedicated box and wrong on the one machine most people start with. With Node:OnDemand the local Ollama and each tool take the card one at a time: a newcomer waits for the holder's in-flight work, the holder is released — only the models this node loaded, and a tool's workers stopped outright so the CUDA context goes with them — and a service quiet for ReleaseAfterSeconds lets go even if nobody asked. Off by default, and a node that does not set it is byte-identical to v3.49. The price is a cold start on every switch, and that is stated rather than tuned away. How it got here is below.

InferHub 3.15 made a two-minute image behave like a two-minute job. v3.14 put Stable Diffusion on the fleet behind OpenAI's Images API; the thing it did not have was any way to watch one. Now POST /api/images/jobs returns an id and a place in line, GET …/events streams queued → running(step 7/28) → succeeded over SSE, GET …/content/0 collects the bytes once, and DELETE changes your mind — cooperatively, so the worker keeps its weights and the next caller does not pay for your cancel. Results live for five minutes and, unless you turn on v3.24's durability, nothing touches disk: no temp file, no cache directory, no URL. /v1/images/generations is unchanged for anyone who never reads this paragraph — internally it became "submit a job and wait for it", so both surfaces queue in the same line and are metered by the same code. Zero new dependencies, no new model, and a deployment that changes no config behaves exactly as it did on 3.14.

What's next. The Qdrant track is finished: a connector (v3.1), server-side hybrid fusion (v3.2), and production knobs plus a migration tool (v3.3) — all three at zero new dependencies. v3.4 through v3.7 turned to the node — supervising its own backend, serving the hub's API, retrieving for itself, and shipping as one container that carries its own Ollama and uses your card, your CPU, or neither. v3.8 started the current track: routing asks what a node can do, not just what it holds — the seam a node needs before it can run something other than a language model. v3.9 is that something, a tool runtime that lets a node drive supervised child processes (Python, in practice) with two consents, a restart budget and no new dependencies; v3.10 is the first real tool, Whisper and Piper behind the OpenAI audio API on a :tools image; and v3.11 turns the same track outward, letting the hub configure the fleet within what each operator already allowed, and v3.12 uses that seam for the thing it was shaped for: a corpus on every node, assigned from one place, with ownership recorded so the hub cannot overwrite what it granted. v3.13 closes the track: the console, the Prometheus series and the docs pass that make six releases' worth of capabilities, tools, audio, profiles and per-node corpora operable by someone who did not write them. v3.14 opened the image track — text to image on OpenAI's Images API, a fifth :diffusion image, and the capability seam carrying a whole new modality with no protocol change at all — and v3.15 gave it a clock: async jobs with per-step progress, cooperative cancellation and results that live for five minutes and are read once. v3.16 makes it a catalogue: six models at the time, quantization that fits a 20B transformer and its 8.3B text encoder on one consumer card, a VRAM budget you declare rather than one we guess, hub-driven weight pulls, and a licence consent per model — because two of them are not ours to accept for you. v3.17 ships the model the track was asked for: 360° panoramas from a rank-128 LoRA over that 20B base, with the projection declared rather than guessed from an aspect ratio, a measured seam, and a hand-written WebGL viewer. v3.18 lets you change a picture rather than only make one: inpainting, img2img and variations as their own capability kind, with the mask convention converted where a pixel may actually be read and a strength knob that bills the steps it ran. v3.19 closes the track the way v3.13 closed the last one: one page that shows what each node can generate, what it is generating right now and at which step, what it refused and why, and a cancel button — plus the series to alert on and a walkthrough somebody can follow top to bottom.

Six releases, a whole new modality, and still zero new dependencies: PyTorch is a child process, not a package.

v3.20 through v3.22 changed nothing InferHub does, and said so. v3.20 split a 2 984-line CLAUDE.md — roughly 64 000 tokens loaded into every agent session before a question was asked, 88% of it decision history — seven ways along the directory tree, because that is what the loader keys on, and split one 1 243-test project into four suites you can run one of. ContextContractTests holds every decision block to existing exactly once, under a line budget, because prose has no compiler; the same pass found CI one path away from going green having run zero tests. v3.21 is the one with a feature in it: an upload past 25 MB streams through the hub, 64 KB at a time into the node's scratch file, instead of being buffered in it — off by default, and a streamed job is never retried on another node, because a client's socket cannot be rewound. v3.22 gave the build briefs the treatment the instructions got: a brief is written the day its phase starts, and the argument behind a decision lands beside the code it governs rather than in a plan that is two phases stale by the time anyone reads it.

v3.23 lets you close that seam — and only if you ask. We measured the flaw for six releases and refused to fix it, because a repair nobody asked for is a second pass they did not watch and would be billed for. What changed is the asking: an operator permits a mechanism, a request chooses one, and the cheap one costs no steps at all.

v3.24 lets a job survive the hub that made it — if you say so. A restart used to turn a job id from thirty seconds ago into a 404 byte-identical to one that never existed. Writing the job down is a file write; the release is the list of things durability may not do — extend retention, survive a read, resume your job (nothing durable holds a prompt), or go in the database.

v3.25 makes the fleet render video — POST /v1/videos, and it is OpenAI's own API rather than one we invented. Phase 47 built InferHub's async image surface because OpenAI has no asynchronous Images API to adopt; it has an asynchronous Videos API, so this release adopts it and adds nothing. One model — wan-t2v-1.3b, Apache-2.0, 480p, 2–5 seconds — over the same queue, the same cancel, the same read-once retention and the same optional durability v3.15 and v3.24 already built. Still zero new PackageReferences: the encoder is a static ffmpeg binary inside a Python wheel, reached through the same child process as everything else.

v3.26 gives that surface a catalogue — three models instead of one. wan-t2v-14b-720p at 720p and cogvideox-2b at 8 fps are what make the fields a catalogue of one could not test: fps is now required and its old fallback of 16 is gone, because encoding CogVideoX's 49 frames at twice their rate is not an error — it is a clip that plays at double speed. The 14B entry is also the first recipe this project ships that does not fit a 24 GB card, which is the VRAM gate working: such a node never declares it, so nobody meets the ceiling mid-render.

v3.27 made the video track visible, and v3.28 found it had never worked. v3.27 gave the console a Video panel that drives /v1/videos — the routes an SDK calls — plus exactly one client-scoped listing route of its own; it put a media label on the image series instead of minting a second family, reported why a video recipe is not offered, and made VideoSecondsPerDay a gate beside MegapixelStepsPerDay. v3.28 shipped no feature: it was the day every published image was pulled onto one box with a real card and driven, and it found that no clip could be generated through any of the three releases before it. The diffusion manifest never declared video, so the operator's ceiling — correctly — threw the worker's claim away; behind that, ftfy was missing from the image, and so was peft, which meant qwen-360 could not load at all. Every test passed, because the fixtures declared their own manifests. The first clip ever rendered and watched was 81 frames at 832×480 in 378 s, and Dispatcher:TimeoutSeconds still defaults to 300 — documented with the measured figures rather than raised, because one number covered every kind of job (until v3.54 split it per capability — Dispatcher:Deadlines). A green suite describes the fixture; only the artefact describes the release.

v3.32 speaks Gemini's own :generateContent, and reading the docs on the day found it is now the legacy one. Type: "gemini" is the third dialect behind the seam — still hand-rolled, still zero new packages. Google's current documentation recommends their newer Interactions API for new work and says generateContent "remains fully supported" with no removal date; this release targets it deliberately, because Interactions' value is the agentic, stateful surface InferHub is not, and :generateContent is what Vertex AI and every gateway in between also speak. What the native dialect buys is the numbers: promptTokenCount already contains the cached tokens here, where Anthropic reports them beside the input count — the same arithmetic would be wrong in opposite directions, so neither is adjusted — and thinking tokens are billed as output while sitting outside the answer's count, which is why ThinkingBudget exists rather than a sum this hub invented. Streaming usage is cumulative, so counts are taken and never summed, for the second vendor running; ?alt=sse is not optional (without it the endpoint answers with a JSON array and an SSE reader simply waits); and a prompt the safety filters refuse arrives as a 200 with no candidates, which is the third success status in this track that is not one. v3.32.1 adds a fourth, found by running the published image: a streaming response that is not SSE at all — a block delivered before the stream starts, or the JSON array you get when alt=sse does not reach the vendor — had every line skipped and produced an empty answer marked done: true. It now throws and names which of the two it was.

v3.31 speaks Anthropic's own /v1/messages, not their OpenAI-compatibility endpoint. Type: "anthropic" is the first real second dialect behind the seam — hand-rolled over HttpClient, still zero new packages — and the reason to write one is the number you get back: Anthropic reports four token counts, the compatibility layer flattens them into two, and the first question anyone asks is why /api/admin/usage disagrees with the invoice. Four wire facts were read from their docs on the day rather than recalled, and two of them contradicted the first draft of the plan: message_delta usage is cumulative (so counts are taken, never summed — message_start already carries output_tokens: 1), and there is no [DONE] sentinel — the stream ends at message_stop. max_tokens is required by the vendor and has no Ollama equivalent, so it is declared per provider; an unknown SSE event is skipped rather than fatal; and an error event mid-stream is raised rather than ending the response as a 200 that looks finished.

v3.30 adds OpenRouter, and it cost no new dialect — which is the point. Type: "openrouter" speaks the same OpenAI wire format the seam already spoke, so what the type actually buys is the part that was never the dialect: a base URL you need not type, a model map checked at boot against OpenRouter's vendor/model id shape, and attribution headers that are absent unless you set them, because they list your deployment on a vendor's public rankings and that is not a thing this hub volunteers on your behalf. Two bugs fell out of reading their docs, and both were live for every OpenAI-compatible upstream: an error.code that is a number instead of a string made every failure arrive as a wall of raw JSON, and an error that happened mid-stream ended the response quietly — 200, and it looked finished.

v3.29 gives the cloud upstream a name — and lets there be more than one of them. Named providers turns the single anonymous Fallback: upstream into a map: OpenAI with one key, OpenRouter with another, a vLLM box on your own network with none, each with its own models and its own trigger. The consent model is untouched — a model absent from every ModelMap is still never sent anywhere — and one model may be claimed by exactly one enabled provider, with a second claim failing startup by name, because which vendor receives a prompt is not something this hub will decide by configuration ordering. A deployment that keeps its Fallback: block behaves byte-identically, header included. This is the first of eight releases that end with Anthropic and Gemini speaking their own dialects and a request being routed to a provider rather than falling into one; zero new PackageReferences, and there will be none in the other seven either.

v3.33 is what the four before it were building towards: a provider becomes a routing target. Until then a vendor was only ever asked after the fleet failed, so "serve smart from Anthropic while my own boxes stay busy" was not a sentence the config could say. Policy is Trigger with two more values — prefer asks the provider first with the fleet as backstop, only asks it always — ModelPolicy overrides it per model, and a Policy and Trigger that disagree fail startup naming both. X-InferHub-Provider steers one request and can only ever narrow: a provider id serves only if it already claims the model, node keeps that one prompt off every vendor, and a wrong steer gets the same sentence whether the id is unknown, parked or mapping something else, so a key cannot enumerate your vendors by probing. The backstop is the policy's answer, not the error's: a failing prefer may fall back to a node, a failing only is a 502, because answering from different weights than the caller asked for, silently, is the one failure that looks like a success. Provider-claimed models now appear in /api/tags and /v1/models with a null digest and size and no vendor named — until then a mapped model was one a client could call and could not discover.

v3.34 puts five releases of provider work on a page. The Cloud providers panel shows every place a prompt can go — policy, credential, claimed models, dispatches, failures and the vendor's own sentence about the last one — and it is the one panel that stays visible when empty, to say nothing leaves your machines. inferhub_provider_info exists before any traffic, which is the only way "no vendor configured" and "a vendor configured that has served nothing" stop being the same silence; inferhub_provider_refused_total carries no label at all, because a label taken from a caller's header is cardinality anybody with a key can mint. The vendor's error text is treated as content: held once per provider, in memory, admin-gated, never a label.

v3.35 gives the node the three dialects the hub got in v3.30–v3.32. Backend:Type is now ollama, openai, openrouter, anthropic or gemini, composed from the same InferHub.Shared code rather than a second implementation, so a GPU-less box becomes a private, authenticated, RAG-capable front end to a vendor, meshed or solo. One node is one upstream and the node never grows a router — the hub chooses, the node serves. An Anthropic-backed node declares chat and not embed, so an embedding request is a 503 at the hub before the hop rather than a failure inside a job; a vendor-typed node with no allowlist refuses to boot, because OpenRouter lists 419 model ids and a node reporting the catalogue would tell the router it can chat with an image model.

v3.36 tells the hub what the node already knew. Since v3.4 a node whose Ollama had died reported zero models, which unrouted it and turned a request for llama3 into 404 model not found — confident, specific, and sending an operator to pull weights already on the disk. Now the node's healthy / unreachable / wedged verdict rides the heartbeat: the model stays listed, the request is a 503 naming the backend, and recovery is the next heartbeat. Ollama:Supervisor:Watch is on by default and restarting still is not — asking a server whether it is alive needs no consent; bouncing one does. Driving the published image then showed the fix holding for six seconds before a timed empty model report brought the 404 back. v3.36.1 made the hub hold an empty list from a node that has declared itself sick; v3.36.2 fixed the root, because the empty report usually arrived before the verdict: a backend that could not be asked now returns null rather than an empty list, since a failure reported as data is indistinguishable from a box whose weights were deleted. Ninety seconds of the right answer on the image, where v3.36.0 gave six. That closes the eight-release provider track, bar its verification day.

v3.37 answers while it is still speaking. stream_format — OpenAI's own field, both values — streams wav and pcm as Piper makes them, split at 16 KiB so every frame fits under SignalR's 32 KB default, because exceeding that limit kills the connection rather than the message and v3.10.0 shipped dead on arrival for exactly that reason. On the published image the first byte left in 0.205 s against 1.228 s buffered, for the same total: nothing got faster, the first byte just left earlier. A streamed wav carries 0xFFFFFFFF in its length fields; the draft claimed ffprobe would report a nonsense duration, running the image showed it reads the correct one from the byte count, and the sentence was corrected everywhere before the blog post went out.

v3.38 lets a node run Postgres as its own vector provider. The refusal it replaced was right about the wrong thing: Npgsql could never go into InferHub.Shared, which is still an empty <Project> with zero packages — so PostgresVectorStore moved into a new InferHub.Shared.Postgres that both hosts reference, one store rather than two, and InferHub.Node.csproj took its first deliberate PackageReference and says so. There is no credentialRef path for it, because a connection string already carries its own password. v3.38.1 followed the same day: none of the five Dockerfiles copied the new project in, which a from-source build never notices.

v3.39 lets an operator say the sentence routing could not: keep this box answering chat, just not with the 70B somebody staged on it for testing. POST /api/admin/nodes/{id}/models/{model}/disable and …/enable change only what the node declares — nothing on disk, no restart, no un-pull — so the router needed zero changes to honour it. Re-enabling runs a VRAM precheck estimated from the disk size Ollama reports and blocks when it does not fit; it is the one refusal here that ships with an override, force=true, because it gates on a guess rather than a fact somebody declared, and the response always says which of the two happened. Collection assign/unassign got the same treatment, with a 409 naming the owner rather than a silent re-parent. (It shipped without release notes of its own, and v3.40's say so rather than backfilling one from memory.)

v3.40 fans one query out to several corpora — hub-owned and node-owned, mixed — through POST /api/retrieve/federated. Each name runs the exact single-collection path, the ranked lists are fused by Reciprocal Rank Fusion keyed on (collection, id) so two corpora's chunk "1" never merge into one scored entry, and sources[] names every collection's outcome: ok, timeout, unavailable or not_found. "Three matches, one source timed out" is a real answer; hiding the timeout would make partial coverage look like completeness.

v3.41 and v3.43 teach the fleet to flip that toggle itself. The auto-scaler re-enables a model drawing refusals on a node that already holds it disabled, through the same precheck a human's call runs and never with force=true; v3.43 adds the other direction, disabling a (node, model) pair idle past IdleMinutes unless it is the last node routing that model, and never in the first MinUptimeMinutes after a restart, so silence after boot is not read as idleness. The first trigger v3.41 wrote read the cloud-fallback counter, and reading the routing code before tagging showed it could never fire: burst asks whether a node holds a model, and a disabled model is still held. The signal it needed — the 503 naming the missing capability — was logged and counted by nothing, so v3.41 added the counter. Both directions are off by default and dry-run when on: the first thing to turn on is the log line, not the write.

v3.42 lets a node-owned corpus outlive its node. An admin assigns a standby; the hub relays the primary's snapshot and every later write to it without ever holding a copy itself, and after a grace period of confirmed absence — not a reconnect blip, not a redeploy — the standby is promoted through the same profile assignment an admin would use. There is no automatic failback, and it covers the local provider only.

v3.44 lets a tool manifest's command be keyed by platform. Investigating "Windows tool workers" found the runtime was already plain System.Diagnostics.Process; the whole gap was one JSON field. A manifest with no branch for this OS is refused by name and the node stays up. The shipped whisper.json and piper.json gained no Windows branch, because that would be a claim about a Windows venv nobody built.

v3.45 ships the reranker the IReranker seam was written for in v2.6. Retrieval:Rerank=cross-encoder routes to sentence-transformers' CrossEncoder over BAAI's bge-reranker family — four models, one new rerank manifest, on the existing :tools image — instead of a chat model asked to score passages it was never trained to rank. It is the first tool worker that moves no file, so it needed no new route: POST /api/tools/{capability} already spoke JSON in, JSON out. Running the worker for real deadlocked it on the first request, every time — a lazy import of numpy's native extension on a request thread while the main thread sat in a blocking read — and the fix was to import once at boot, so ready means ready. v3.45.1 followed within hours: both options validators still refused cross-encoder, so a node configured exactly as the docs said would not start. Pulling the published image was the first thing that ever went through ValidateOnStart.

v3.46 pushes what /metrics already serves to an OTLP collector — Grafana Cloud, Honeycomb, Datadog, the reference otelcol — for an operator with no Prometheus server to scrape. It reads the scrape's own exposition text back and posts it as OTLP/HTTP JSON, so there is one source of truth rather than two places to remember a metric; the OpenTelemetry SDK was declined for the protobuf and gRPC it brings. A failed push is dropped, not queued: a stale number sent late is the past reported as now.

v3.47 adds the one ceiling a coordinator cannot even see. Node:ResourceLimits caps how much of its own CPU and GPU a box lets itself be routed to consume. Every other node-side limit is one a hub profile can narrow further; this one is not on the wire, so a coordinator — trusted or compromised — cannot set it, raise it or read it. Over MaxCpuPercent or MaxGpuPercent for SustainedPolls consecutive polls, a meshed node withdraws from new placement exactly the way an unhealthy backend does and a solo node answers 503 + Retry-After; nothing already running is touched. HardCpuCapPercent is a real Windows Job Object rate limit on tool workers, and the node serves its own loopback-guarded page at /api/admin/resource-limits whose saves take effect within one poll, no restart. (v3.47.0 failed its own context-contract test on a mistyped pointer; v3.47.1 is the one to run.) The published-image check found a methodology trap rather than a bug: on Docker Desktop, load on the Windows host never registers inside the container, because its /proc/stat belongs to the VM.

v3.48 makes "this is not a sandbox" optional. A manifest may name "sandbox": { "mode": "bubblewrap" }, and its worker gets read-only access to its own tree and the base OS, read-write access to the scratch directory and nothing else, and no network unless network: true. Linux only, with no unsandboxed fallback: silently running unsandboxed what an operator asked to sandbox is the one failure it exists to prevent. Running it for real found three things the bwrap manual does not warn about — merged-usr symlinks, bind order, and --die-with-parent SIGKILLing workers because Linux ties it to the thread that forked, which .NET recycles — and the last was dropped rather than papered over, leaving a named orphan gap on an unclean node exit. No shipped manifest is sandboxed yet; what changed is what a worker can see and reach, not what it can do.

v3.49 adds a second speak engine, and with it a sixth image. beleata74/bg-tts-v5 is a 250.8M-parameter Bulgarian model over NVIDIA's NanoCodec — nothing Piper's .onnx shape fits — served from inferhub-node:tts-bg as bg-tts-v5-spk0 and bg-tts-v5-spk1: two model names over one loaded checkpoint, because a model name that pins the speaker cannot drift the way a voice field could. It is its own image rather than a flag, because NeMo's pinned torch beside :diffusion's is two CUDA builds asking whose pin wins. The 2.9 GB checkpoint is placed by hand and only the small codec is fetched; streaming and speed are each refused by name. Running the vendored upstream code found two bugs its own example never hit, the reranker's loader-lock one import deeper, and NeMo's logger writing a bare line onto the worker's protocol stdout.

v3.50, at the top of this section, gives a desktop its card back — and its first draft released Ollama by unloading everything resident. The box it was about to run on had a 22 GB model loaded through that same Ollama by another program. So the node records every model it sends a request for and unloads only those, which is the difference between a mode that shares a desktop and one that merely occupies it.

What is still open is mostly a list the releases wrote about themselves. Phase 68, the provider verification day, is parked awaiting vendor keys: every dialect from v3.29 to v3.35 has been driven against stubs and recorded payloads, and the checks that need one real key per vendor have not been started — until they are, "the node speaks Anthropic" is a claim about translation, not about a conversation anybody has had. Five releases name a live drill still owed: a wedged backend end to end (v3.36), a two-node standby-and-docker kill (v3.42), scale-in on a real idle fleet (v3.43), the cross-encoder through a full coordinator-plus-node retrieval pipeline (v3.45), and a throttled node's traffic actually rerouting (v3.47). (v3.50's, a real CUDA worker switching under Node:OnDemand, was run on v3.52's :all image: Ollama, sd15 and Whisper took a 3090 Ti in turn.) A few gaps are design rather than debt: no shipped tool manifest runs sandboxed until its cache directories are audited, and seccomp and UID remapping are out of scope; a standby covers only the local provider; the OTLP exporter skips histograms. Beyond those, the two items the previous list carried still stand — active-active multi-coordinator load sharing, and a fourth vector backend (Milvus, Weaviate), which is the same shape as the third and will ship on real demand rather than for the comparison matrix.

Quick start

Docker (recommended)

cp deploy/docker/.env.example deploy/docker/.env    # set three keys
docker compose -f deploy/docker/docker-compose.yml up -d

In Docker there is no loopback exemption. Requests from your host arrive over the bridge network, not loopback, so the API keys are mandatory — unlike the from-source path below. See deploy/docker/README.md.

Images are published to GHCR for linux/amd64 and linux/arm64: ghcr.io/dev-art-solutions/inferhub-coordinator and .../inferhub-node.

Which image do I pull? (v3.13)

Four artifacts with no chooser is how somebody pulls 6 GB to run a 340 MB workload, or pulls the small one and wonders where the audio went. All tags are under ghcr.io/dev-art-solutions/.

Three rules of thumb that save the mistake each way:

• Do not pull :tools for chat. It is :ollama plus ~1.5 GB of Python wheels you will never load. Every image above it does chat identically.
• Do not pull :ollama to point at an Ollama you already have. The bundled one would sit idle next to it — or worse, fight it for a port. Use the plain image and set Ollama:Endpoint.
• :diffusion is the one image that does not stack, and that is deliberate: it has no Ollama, no Whisper and no Piper in it. A card running a diffusion pipeline has no room for a chat model beside it, so bundling one would ship a combination we would then have to tell you not to use. Want both? Run two containers and let the coordinator route image to one and chat to the other — that is what capability routing is for. Unless you have one card: two containers on one card are two nodes, each sure the card is its own. That box wants :all (v3.52+), one node whose services take turns.

Whichever node image you choose, mount a volume at /data. Model weights, the node's stable id, tool scratch and any corpus live there; without it every docker run re-downloads gigabytes.

From source

# On the always-on host (no GPU needed)
dotnet run --project src/InferHub.Coordinator

# On each GPU machine (with Ollama already running locally)
dotnet run --project src/InferHub.Node

# From anywhere, talk to it like Ollama (remote calls need a Bearer token)
curl http://your-coordinator:5080/api/chat \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"model":"llama3","messages":[{"role":"user","content":"Hello!"}],"stream":false}'

Just a node — solo mode (v3.5+)

Both of the above run two processes so that one client can reach one backend. On a single machine that is all ceremony and no routing, so since v3.5 the node can serve the same API itself — no coordinator, no enrollment secret, no internet.

dotnet run --project src/InferHub.Node \
  --LocalApi:Enabled=true --Coordinator:Enabled=false

# before — the fleet
client = OpenAI(base_url="http://hub.example:5080/v1", api_key=KEY)
# after — just the node
client = OpenAI(base_url="http://localhost:5081/v1", api_key=KEY)

That is the whole migration. Same request bodies, same responses, same streaming, same errors, both dialects. Full details, and what solo mode deliberately does not do, are in Solo mode.

One container, with or without a GPU (v3.7+)

The same thing again with nothing installed on the host — the bundled image carries Ollama inside it:

docker run -d --name inferhub --gpus all \
  -e LocalApi__Enabled=true -e Coordinator__Enabled=false \
  -e LocalApi__ApiKeys__0="$INFERHUB_API_KEY" \
  -v inferhub:/data -p 5081:8080 \
  ghcr.io/dev-art-solutions/inferhub-node:ollama

docker exec inferhub ollama pull llama3.2

Drop --gpus all and it runs on the CPU instead. See A node and its Ollama in one container.

Solo mode — just the node (v3.5+)

InferHub had exactly one shape until v3.5: a coordinator somewhere always-on, and nodes on GPU machines dialling out to it. That shape is right when you have a fleet and it is overhead when you have one machine — and one machine is how most people start.

// src/InferHub.Node/appsettings.json
{
  "Coordinator": { "Enabled": false },
  "LocalApi": {
    "Enabled": true,
    "Urls": "http://localhost:5081"
  }
}

A solo node is the hub's formatting layer sitting directly on the node's executor, with the routing layer removed — the hub translates a request, routes it, queues it, dispatches it over SignalR and formats what comes back; solo does the first and last and skips the middle, because there is nothing to route to. Both ends were already shared code, which is why tool calls, vision and both dialects work here without being reimplemented.

What it serves

/api/status deliberately does not return the hub's document with zeros in the fleet fields: a dashboard reading nodesEvicted: 0 from a process that has no concept of nodes is worse than one that gets a 404 for a key that was never there.

Retrieval on a standalone node (v3.6+)

A machine with a folder of documents and one GPU is the deployment that most wants RAG, so since v3.6 a standalone node ingests, indexes and grounds its own answers — the same headers, the same augmented prompt and the same X-InferHub-Sources citations as a coordinator, because it is the same code rather than a second copy of it.

"Coordinator": { "Enabled": false },
"LocalApi": {
  "Enabled": true,
  "Retrieval": { "Enabled": true, "DefaultEmbeddingModel": "nomic-embed-text" }
}

# ingest
curl localhost:5081/api/collections/handbook/documents \
  -H 'content-type: application/json' \
  -d '{"id":"leave-policy","text":"Employees accrue 25 days of annual leave each year."}'

# see what retrieves, before you trust it
curl localhost:5081/api/collections/handbook/search \
  -H 'content-type: application/json' -d '{"query":"how much annual leave?"}'

# ground an answer
curl localhost:5081/api/chat -H 'X-InferHub-Retrieve: handbook' \
  -H 'content-type: application/json' \
  -d '{"model":"llama3","messages":[{"role":"user","content":"how much annual leave?"}],"stream":false}'

That adds /api/collections/{c}/documents (ingest, list, chunks, delete), the /api/collections/{c}/search playground, the /api/vector/{c} data plane, and X-InferHub-Retrieve on both dialects. Vector, keyword and hybrid modes all work, as does the optional LLM reranker.

It requires Coordinator:Enabled=false, and says so by refusing to start otherwise. A node in a mesh already holds vector replicas derived from its coordinator; giving that same process an authoritative corpus of its own would put two sources of truth under one collection name — a locally ingested document invisible to the fleet, and a hub that will happily overwrite it. There is no safe configuration of that, so it is not offered, and it fails loudly rather than quietly switching grounding off.

No PDF. The PDF text extractor ships with the coordinator only, so a PDF upload gets a clean 415 telling you to convert the file or use a hub — never a silently bad extraction that fills your corpus with plausible nonsense. No external vector databases either: the local store is the only provider on a node. The corpus lives in LocalApi:Retrieval:DataDirectory (in Docker, /data/retrieval — mount a volume or it is ephemeral).

Two more things it deliberately will not do

No admin API and no console. There is one node and you are sitting at it. Model management stays hub-driven; in solo mode it is ollama pull in the terminal you already have open.

It is not a second kind of coordinator. A solo node serves its own clients and nothing else. Nothing about the fleet changes: the coordinator gained no code, meshed nodes behave exactly as before, and the outbound-only rule that lets a GPU box sit behind NAT with no inbound rule is untouched — solo mode adds a surface for your own clients, which is why it needs no hub at all.

Auth, and the one place it refuses to boot

Off by default, and loopback when on. Loopback callers need no key (matching the hub's Auth:RequireAuthForLoopback), so local curl just works.

A non-loopback address with no keys fails startup, naming the key. That is stricter than InferHub usually is about somebody else's network, and the asymmetry is intentional: an unauthenticated inference endpoint hands arbitrary compute on your GPU to anyone who can reach the port, and the first sign of it is a bill or a melted card. LocalApi:AllowAnonymous=true is the explicit override for a trusted network, and it warns on every boot.

This bites in a container, by design — the image binds a wildcard, so a containerised solo node needs a key:

docker run -e LocalApi__Enabled=true -e Coordinator__Enabled=false \
           -e LocalApi__ApiKeys__0=your-key \
           -e Ollama__Endpoint=http://host.docker.internal:11434/ \
           -p 5081:8080 ghcr.io/dev-art-solutions/inferhub-node

Concurrency

Node:MaxConcurrency has always been advisory — a number the coordinator's router respects. In solo mode nobody is respecting it, so it is enforced locally: over the cap a request waits up to LocalApi:MaxWaitSeconds and then gets 503 + Retry-After, the same status and header as the hub's queue, so existing client retry logic behaves identically. Unset means unbounded, exactly as it does today.

Both at once

Coordinator:Enabled and LocalApi:Enabled are independent. A fleet node with the local API on is legitimate — useful for curling a node directly while debugging. Both off is a startup failure naming both keys: a node that neither joins a mesh nor serves anyone is a typo.

A node and its Ollama in one container (v3.7+)

Solo mode removed the coordinator. This removes the other half: the Ollama you had to install on the host, reach through host.docker.internal or 172.17.0.1, and keep alive yourself.

docker run -d --name inferhub --gpus all \
  -e LocalApi__Enabled=true -e Coordinator__Enabled=false \
  -e LocalApi__ApiKeys__0="$INFERHUB_API_KEY" \
  -v inferhub:/data -p 5081:8080 \
  ghcr.io/dev-art-solutions/inferhub-node:ollama

docker exec inferhub ollama pull llama3.2

Then point any OpenAI client at http://localhost:5081/v1.

Three modes, one image

None of them refuses to start, and the node says in its first log lines which one it is in.

The third is worth explaining: /api/vector/{collection} takes client-supplied vectors, so a node with no inference process is still a complete vector store. It reports zero models, which is the honest answer — a chat request fails cleanly instead of hanging. Ingesting documents does need an embedder, so in that mode either bring your own vectors or point Backend:Type=openai at an embedding upstream elsewhere. And it is still the 4 GB image: if you only ever want this, the plain inferhub-node image does solo retrieval identically at 340 MB.

Did it get the card?

The node answers that in its first log lines, both ways:

info: CUDA: 1 device(s) visible to this process — NVIDIA GeForce RTX 3090 Ti.
info: CUDA: no devices visible to this process; inference will run on the CPU.
      In a container, pass '--gpus all' to use a card.

and on /api/status as a gpu block. For which device a loaded model actually ended up on, docker exec inferhub ollama ps is the ground truth — that is Ollama's own accounting and it is not something InferHub second-guesses.

Ollama:RequireGpu=true turns a missing card into a startup failure naming --gpus all. It is off by default, including in this image, because CPU is a supported mode rather than a misconfiguration. Turn it on for a node whose whole purpose is the card, so a --gpus flag falling out of a unit file is not a silent fiftyfold slowdown.

What is in it, and what is not

• ~4 GB (the plain inferhub-node image is ~340 MB and does not change). amd64 only, and NVIDIA only — no ROCm, no Intel, no Apple, and arm64 would mean Jetson-specific bundles.
• No model is baked in and none is pulled at boot. docker exec … ollama pull is the interface. Mount a volume at /data or every docker run re-downloads them.
• Ollama's own port is not published. The container's surface is InferHub's API, which requires a key. -e OLLAMA_HOST=0.0.0.0 -p 11434:11434 is yours to decide.
• Every OLLAMA_* variable passes through — OLLAMA_KEEP_ALIVE, OLLAMA_NUM_PARALLEL, OLLAMA_FLASH_ATTENTION and the rest are one -e away.
• The node keeps its Ollama alive, using the supervisor: it starts it at boot, probes it, and restarts it if it dies or wedges. That matters more in a container than on a host, because nothing else in there is watching.
• On Windows with Docker Desktop this works through WSL2 — note that /dev/nvidia* does not exist there, so InferHub detects the GPU by loading the driver rather than by looking for device nodes.

It is also a good mesh node: set Coordinator__Url instead of Coordinator__Enabled=false, and the hub can pull models into it from the console, because the Ollama it manages is genuinely its own.

docker compose -f deploy/docker/compose.ollama.yml up -d

What a node is for (v3.8+)

Until v3.8 a node advertised a list of model names, and the coordinator routed on that alone. That is a routing key with one dimension, and it quietly assumes every model on a node does the same kind of work. It does not: a box holding only nomic-embed-text was a perfectly good candidate for a chat request naming that model, and the error arrived from the backend, after a dispatch.

Since v3.8 a node also declares capabilities — what it can do — and the unit of routing is the pair (capability, model):

curl -s localhost:5080/api/status | jq '.capabilities, .nodes[] | {name, capabilities}'
# { "capability": "chat",  "nodes": 2, "models": ["llama3.2", "qwen2.5"] }
# { "capability": "embed", "nodes": 3, "models": ["llama3.2", "nomic-embed-text", "qwen2.5"] }

curl -s localhost:5080/v1/models | jq '.data[] | {id, capabilities}'
# { "id": "nomic-embed-text", "capabilities": ["embed"] }

Nothing is guessed. Ollama does not say what a model is for, and inferring it from the name would be a lookup table that is wrong for somebody. A node declares chat + embed over everything its backend reports, and the one thing it cannot work out for itself — that this box is for embeddings — is one line of config:

// src/InferHub.Node/appsettings.json, or Node__Capabilities__Disabled__0=chat
"Node": { "Capabilities": { "Disabled": ["chat"] } }

That node then never receives a chat job. A client that asks for one anyway gets a 503 with Retry-After naming the capability — not a 404, because the model is there and this is a fact about the fleet right now, the same category as every node being busy. A model that genuinely is not on the fleet is still the 404 it has always been.

In solo mode the same key is enforced by the node itself, with the same status and the same header, because one key must not mean two different things depending on how the node is deployed. (Its own corpus is exempt: solo RAG still embeds with embed disabled, since the node's own documents are not somebody sending it work.)

A v3.7 node against a v3.8 coordinator is routed exactly as before — no declaration means chat and embed over everything, which is precisely the old behaviour, so a fleet can be upgraded one box at a time.

This is the seam the next few releases need: a node that can run a speech model, or anything else that is not a language model, has to be able to say so before anything can be routed to it.

Tools on a node (v3.9+)

A tool worker is a child process a node starts, supervises, talks to over a line protocol, and restarts when it dies. It is how a node does work its inference backend cannot — transcription, speech, OCR, whatever you write — and it is off by default.

// the manifest, in Tools:ManifestDirectory
{
  "id": "whisper",
  "capabilities": [ { "kind": "transcribe", "models": ["whisper-small"] } ],
  "command": ["/opt/inferhub/venv/bin/python", "-u", "/opt/inferhub/tools/whisper_worker.py"],
  "env": { "HF_HOME": "/data/tools/hf" },
  "maxWorkers": 1
}

// and the two consents on the node
"Tools": { "Enabled": true, "Allowed": [ "whisper" ] }

The node then declares transcribe as a capability, the coordinator routes (transcribe, whisper-small) to it, and it is reachable the same way on a hub and on a solo node:

curl localhost:5080/api/tools/transcribe -H "Authorization: Bearer $KEY" \
  -F model=whisper-small -F file=@meeting.m4a

python/README.md is the worker author's document — the protocol, the manifest reference, and what the node does when a worker misbehaves. python/inferhub_worker/ is a ~150-line reference implementation you copy or vendor; it is deliberately not a package.

command (and workdir) may be keyed by platform instead of a single argv array (v3.44+), for a node running bare-metal or as the Windows service rather than the :tools image, with a tool built for that box:

{
  "id": "my-tool",
  "capabilities": [ { "kind": "echo", "models": ["echo"] } ],
  "command": {
    "linux":   ["/opt/inferhub/venv/bin/python", "-u", "/opt/inferhub/tools/worker.py"],
    "windows": ["C:\\ProgramData\\InferHub\\Node\\tools\\venv\\Scripts\\python.exe", "-u",
                "C:\\ProgramData\\InferHub\\Node\\tools\\worker.py"]
  }
}

One id, both platforms — the branch is picked by the OS the node process is actually running on. A manifest that names no branch for this node's platform is refused by name (logged, skipped, the node stays up); workdir is optional, so a missing branch there resolves to "unset" rather than a refusal. The shipped whisper.json/piper.json are unchanged and stay POSIX-only — they are the :tools image's own manifests, and there is no Windows build of that image.

Why a subprocess and not a library

Because the libraries are Python, and the alternative — Python.NET, CSnakes, an embedded interpreter — is a native binding: it pins the node to a CPython ABI, and one bad import takes the node down, because a segfault in a native extension loaded into this process is not an exception you catch, it is a process that vanishes mid-stream taking every in-flight inference job with it. A child process that segfaults is a log line and a restart. It also means a tool that is a Go binary, an ffmpeg invocation or a vendor's CLI works here for free.

InferHub added zero dependencies for this. The node knows how to start a process, write a line, read a line and kill it. It does not know what Python is.

Opt in twice

Tools:Enabled consents to the feature. Tools:Allowed names the manifest ids that may actually run — a manifest on disk that is not in the list is loaded, logged and never started. The two are not redundant: since v3.11 a coordinator can turn a node's tools and capabilities off through a node profile, and Tools:Allowed is the ceiling it can never raise. A single switch would make "the operator enabled tools" and "the hub may run any tool present on this box" the same consent.

This is not a sandbox (unless you ask for one, v3.48+)

Said plainly, because the alternative is implying safety by listing mitigations.

By default a worker runs as the node's user, with the node's filesystem and the node's network. The node drops its own environment before spawning — a worker gets PATH, HOME, LANG, LC_ALL, TMPDIR, USER, SHELL and whatever the manifest's env names, and never Coordinator__EnrollmentSecret or LocalApi__ApiKeys__0 — and until v3.48 that was the honest extent of the isolation. A tool you did not write and did not read has your box.

An optional sandbox field wraps the worker in bubblewrap (bwrap, in the :tools/:diffusion images already — nothing to install):

{
  "id": "my-tool",
  "command": ["/opt/inferhub/venv/bin/python", "-u", "/opt/inferhub/tools/worker.py"],
  "sandbox": { "mode": "bubblewrap", "network": false }
}

The worker gets read-only access to its own venv/script tree and the base OS, read-write access to exactly Tools:ScratchDirectory and nothing else, and no network namespace at all unless network: true is set. Absent sandbox is unchanged — every manifest that does not write this field runs exactly as before, which is every manifest InferHub ships (whisper.json, piper.json, rerank.json, diffusion.json are deliberately left unsandboxed; auditing their real filesystem needs — HF cache directories outside scratch, voice files — is future work).

Linux-only, and there is no silent fallback: a manifest naming bubblewrap on Windows/macOS is refused at load, by name, the node stays up otherwise. Running the container itself needs two extra capabilities bwrap requires to build its own namespaces:

docker run --cap-add SYS_ADMIN --cap-add NET_ADMIN ... ghcr.io/dev-art-solutions/inferhub-node:tools

Not covered: seccomp syscall filtering and UID-namespace remapping. A sandboxed worker still runs as the node's own uid and can make any syscall the kernel allows — what changed is what it can see and reach, not what it can do. Also not covered: a node that crashes uncleanly (not a normal shutdown) can orphan a sandboxed worker process — --die-with-parent was tried and dropped after it reliably SIGKILLed workers under .NET's threading model (see src/InferHub.Node/Tools/CLAUDE.md, phase-83).

If you want more isolation than this gives you — or want it on Windows — run the tool in its own container and point a manifest at it: a "process" that is docker exec is still a process, and the protocol does not care.

What happens when a tool misbehaves

A tool failure is a failed job, never a failed node. Attachments are capped at 25 MB (Tools:MaxAttachmentBytes, a 413 at the edge), written to a per-request scratch directory that is deleted in a finally — after success and after failure — and never logged.

Uploads larger than 25 MB (v3.21)

Set Tools:MaxStreamedBytes on the coordinator and on a node, and an upload past Tools:MaxAttachmentBytes streams through the hub instead of being buffered in it: the node pulls the bytes over the connection it already opened, 64 KB at a time, straight into the scratch file the worker reads. The hub never holds the body, and raising the key moves Kestrel's own 30 MB request-body ceiling with it — so the 413 you get always names a key you can raise.

It is off by default (0), and with it off a deployment behaves exactly as v3.20 did. Three things are worth knowing before you turn it on:

• A streamed job is not retried on another node. The body has been read and a client's socket cannot be rewound, so a node lost mid-upload is a 502 naming node_lost, and the caller decides. This is a real step down in reliability on a path you opt into by size.
• Send your form fields before the file part. The request is routed before the bytes arrive, so a model that turns up after the file is a 400 that says so. curl -F model=… -F file=@… and the OpenAI SDKs already do this. A solo node has nothing to route and accepts any order.
• It applies to POST /api/tools/{capability} and /v1/audio/transcriptions. The image routes keep the 25 MB cap: an image job is answered before it runs, so its bytes would have to outlive the request that carried them — which is the image store this project has refused three times.

A fleet with no node that can take a streamed upload answers 503 naming that as the reason, rather than quietly falling back to buffering and failing at 25 MB.

v3.9 shipped the machinery and no tool. The test suite drives a real child process that echoes. v3.10 puts Whisper and Piper on it — see below.

Speech in, speech out (v3.10+)

Two endpoints, on OpenAI's audio API exactly, against your own hardware:

# transcription
curl http://localhost:5080/v1/audio/transcriptions \
  -H "Authorization: Bearer $KEY" \
  -F file=@meeting.m4a -F model=whisper-small -F response_format=verbose_json

# speech
curl http://localhost:5080/v1/audio/speech \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"model":"en_US-amy-medium","input":"InferHub can talk now.","response_format":"wav"}' \
  --output out.wav

Every SDK in every language already speaks this, so pointing an existing app at your own GPU is a base-URL change. The same two routes are served by a solo node with no coordinator at all, and by one container:

docker run -d --gpus all \
  -e LocalApi__Enabled=true -e Coordinator__Enabled=false \
  -e LocalApi__ApiKeys__0="$KEY" -v inferhub:/data -p 5081:8080 \
  ghcr.io/dev-art-solutions/inferhub-node:tools

--gpus left off is CPU Whisper — roughly real time for small on a modern core — and the worker's first log line says which one it got.

The six images

Which one to pull, and the three mistakes worth avoiding, are in Which image do I pull?. This is what is inside each.

The first three are unchanged by v3.10. The Python is ~1.5 GB and it is in a layer whether a flag is on or off, so a flag would grow every existing coordinator+node stack for a feature it does not use. An operator on the plain image can still run these workers by installing Python themselves and pointing a manifest at it (python/requirements-tools.txt is what the image installs) — the runtime does not care where the interpreter came from.

Formats

srt and vtt are formatted at the edge from the segments Whisper produces anyway, so a worker author never writes a subtitle timestamp. A format that cannot be produced is a 400 naming the ones that can — never a silent substitution, because a caller who asked for mp3 and got a wav has a corrupted file with a confident content type and finds out in a media player three days later.

Speech that starts before it is finished (v3.37+)

A paragraph is a couple of minutes of audio, and until v3.37 the socket was silent for all of it. Add OpenAI's stream_format and the samples arrive as they are made — one piece per sentence, split so each one fits the wire:

# raw bytes on a chunked body: the shape every SDK already consumes
curl -N http://localhost:5080/v1/audio/speech \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"model":"en_US-amy-medium","input":"One. Two. Three.","response_format":"wav","stream_format":"audio"}' \
  --output out.wav

# or framed as events
  ... "stream_format":"sse"
#   event: speech.audio.delta   {"type":"speech.audio.delta","audio":"<base64>"}
#   event: speech.audio.done    {"type":"speech.audio.done","usage":{...}}

Six things are worth knowing:

• Leaving stream_format out changes nothing. The response is byte for byte what v3.36 returned.
• wav and pcm stream; mp3, opus and flac do not, and asking is a 400 naming the two that do. The others need an encoder running for the length of the request and a chunk boundary that is not a codec frame boundary — a naive split clicks every third of a second.
• A streamed wav has 0xFFFFFFFF in both length fields. The length is not knowable when the header goes out. Measured on the published image: ffprobe reports the correct duration, because a saved file has a byte count and ffmpeg prefers it over the header. What the sentinel costs is narrower — a consumer that trusts that field alone computes ~4 GB.
• The sample rate is on X-InferHub-Audio-Sample-Rate, measured from the audio itself rather than declared. For pcm it is the only place it can be.
• The usage in speech.audio.done is three zeros, and they are true — Piper is a phoneme model and nothing is tokenized. What is billed is unchanged: input characters, also on X-InferHub-Speech-Characters. Putting a character count in a field named tokens would be a four-to-one error on somebody's invoice.
• In a mesh, only nodes running v3.37+ take a streaming request. Older ones keep serving every buffered one; a fleet with none new enough answers 503 naming the version rather than pretending it cannot synthesise at all.

A solo node serves the identical bytes. speech.audio.error — a terminal event carrying the ordinary error envelope when a stream dies after the first byte — is not in OpenAI's schema; it is ours, because closing silently leaves a client unable to tell an ending from a failure.

Models and voices

No weights are baked into the image. Whisper fetches on first use into /data/tools/hf, on the volume, so it happens once rather than once per docker run. That fetch is a reach onto the internet, so it is behind a third opt-in: Tools:AllowModelDownload (default false, set true in the :tools image, because choosing that image is the consent). With it off, a worker that needs missing weights fails the job with the exact pre-fetch command in the message, and the node keeps serving everything else.

No voice is fetched until you name one, because no default voice is right for everyone and a confident answer in the wrong language is worse than a refusal. Since v3.55 you name it and the worker does the rest:

docker run -d -e LocalApi__Enabled=true -e Coordinator__Enabled=false \
  -e LocalApi__ApiKeys__0="$KEY" -e Tools__Speech__Voices__0=bg_BG-dimitar-medium \
  -v inferhub:/data -p 5081:8080 ghcr.io/dev-art-solutions/inferhub-node:tools

The id is looked up in the voice catalogue (python/voices/, /opt/inferhub/voices in the image). That is one file per voice, pinning a commit of rhasspy/piper-voices plus a byte count and a sha256 for each of the voice's two files. The worker downloads in the background, renames the pair into /data/tools/voices only once both match, and offers the voice when it lands. There is no restart and no docker exec. A mismatch is logged with both hashes and nothing is installed. Tools:AllowModelDownload still has the last word: with it off, the log names the two URLs to fetch by hand. HF_ENDPOINT in the manifest's env points the fetch at a mirror.

bg_BG-dimitar-medium is the small Bulgarian voice. bg-tts-v5 (v3.49, the :tts-bg image) is still the better-sounding one with two speakers, and it wants a card. Any other Piper voice works the old way: drop its .onnx + .onnx.json pair into /data/tools/voices and restart. The model name is the file's stem.

curl http://localhost:5081/v1/audio/speech \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"model":"bg_BG-dimitar-medium","input":"Здравейте! Това е InferHub.","response_format":"wav"}' \
  --output bg.wav

None of it is kept

Design rule 7 at its most literal: a transcription request is a recording of somebody's voice and the answer is what they said.

• The hub buffers the upload for the dispatch and drops it. No temp file, no cache.
• The node writes it into a per-request scratch directory that is deleted in a finally, always.
• Nothing containing audio bytes or transcript text is logged, at any level. The line for a transcription carries the model, the duration and the outcome — not the filename you chose.
• The usage ledger gains a duration, never a transcript.

AudioPrivacyTests runs a transcription through a real mesh with a capturing logger and fails if a known phrase from the fixture appears anywhere in the logs or the ledger.

Usage and quotas

Audio has no token count, so metering it as tokens would mean inventing a number. A usage row now carries units and a unitKind — tokens, audio_seconds or characters — and the token fields are untouched, so every existing consumer and every existing row keeps working. Two new client limits go with it:

"Auth": { "Clients": [ { "Id": "acme", "Key": "…",
  "Limits": { "AudioSecondsPerDay": 3600, "CharactersPerDay": 200000 } } ] }

They are separate budgets on purpose: a client whose only limit is TokensPerDay could otherwise transcribe a library for free. Over one is the same 402 + Retry-After to UTC midnight as the token budget.

One transcription at a time

maxWorkers defaults to 1 — a second Whisper process on the same card is two copies of the weights and an out-of-memory error at the worst possible moment. Because routing is per capability (v3.8), a node busy transcribing is still a candidate for chat: "my chat got slow when someone uploaded a podcast" is the failure that prevents.

Not in v3.10, and said out loud: streaming TTS (chunked audio needs a concatenable format and a client-side contract), diarization, speaker labels, voice cloning, and /v1/audio/translations. The last is one flag on the same worker and can land whenever somebody asks.

Text to image (v3.14+)

Stable Diffusion on your own card, through the API your app already calls.

curl http://localhost:5080/v1/images/generations \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"model":"sdxl","prompt":"a lighthouse in a storm, oil painting","size":"1024x1024","n":1}' \
  | jq -r '.data[0].b64_json' | base64 -d > out.png

…the same on a solo node with no coordinator, and the same inside one container:

docker run -d --gpus all \
  -e LocalApi__Enabled=true -e Coordinator__Enabled=false \
  -e LocalApi__ApiKeys__0="$KEY" -v inferhub-images:/data -p 5083:8080 \
  ghcr.io/dev-art-solutions/inferhub-node:diffusion

It is OpenAI's Images API, so pointing an existing app at your own hardware is a base-URL change. The capability seam took the whole modality with no protocol change: image is one more capability kind, and the router already knew how to find (capability, model).

The catalogue (v3.16)

Two of those numbers are the whole point. flux-schnell, qwen-image and qwen-360 do not fit a 24 GB card at bf16 — 33 GB and 60 GB — and nf4 quantization is what makes them one-card models. Both figures are in the table because "Qwen-Image needs 19 GB" and "Qwen-Image needs 60 GB" are both true sentences about different recipes, and a table that gives one is lying to somebody.

Two repositories are gated on Hugging Face, and that is a different thing from the licence. black-forest-labs/FLUX.1-schnell and stabilityai/stable-diffusion-3.5-medium both require you to accept terms on the model page and present a read token, regardless of what their licence says — FLUX is Apache-2.0 and still gated. Accepting a licence in Tools:Image:AcceptedLicenses tells this node it may run a model; a token is how Hugging Face decides whether to hand the weights over. Set Tools:Image:HuggingFaceToken, and note it has to be that key rather than an HF_TOKEN environment variable, because the node clears a worker's environment before spawning it.

Without it the fetch fails with a message that says exactly this, rather than a bare 401:

[diffusion] could not fetch 'flux-schnell': GatedRepoError: this repository is GATED on Hugging Face.
            Accept the terms at https://huggingface.co/black-forest-labs/FLUX.1-schnell with the
            account you are using, then set Tools:Image:HuggingFaceToken to a read token.

The model you send is a recipe id, not a Hugging Face repo id. A repo id is a location: it has a slash in it that every router and metrics label has an opinion about, and it changes when a model is re-hosted. That is not hypothetical — runwayml/stable-diffusion-v1-5 was withdrawn, and sd15 points at where those weights live now.

Recipes are python/recipes/*.json — a repo, a pinned commit sha, the variant, the pipeline class, the aspect buckets, the VRAM figure, the licence and the quantization. Drop one in and restart the tool. A recipe with no pinned revision is skipped and logged, because "which weights were in 3.16.0" has to have an answer.

A picture on a CPU in seconds (v3.56)

lcm-dreamshaper is SimianLuo/LCM_Dreamshaper_v7, a Latent Consistency Model distilled from Dreamshaper 7 (an SD 1.5 fine-tune). It has the same 0.9B UNet as sd15 and needs 4 steps instead of 30. On a CPU-only node (Tools:Image:RequireGpu=false) it is offered with no other setting:

{ "model": "lcm-dreamshaper", "prompt": "a lighthouse on a rocky coast at sunset, oil painting", "size": "512x512" }

Measured on a 64-core box, through the worker's own fetch and load code under the pinned torch 2.9.1+cpu:

768×768 took 36 s at 8 threads. The download is ~4.3 GB: the repo has no fp16 files, and its ONNX copies and single-file checkpoint are never fetched. The licence is declared as CreativeML OpenRAIL-M, inherited from Dreamshaper 7 and SD 1.5, even though the repo is tagged MIT. It is the stricter of the two readings. Both are permissive here, so nothing is gated.

It has no negative prompt, and it would not tell you. An LCM distils guidance into the model, and its pipeline accepts negative_prompt through **kwargs and ignores it. So a request to lcm-dreamshaper that carries one is a 400 naming the recipe.

Editing on a CPU (v3.57)

lcm-dreamshaper also takes /v1/images/edits and /variations, so a CPU-only node can change a picture as well as draw one:

curl http://localhost:5080/v1/images/edits -H "Authorization: Bearer $KEY" \
  -F model=lcm-dreamshaper -F image=@lighthouse.png \
  -F prompt="the same lighthouse in winter, snow" \
  -H 'X-InferHub-Image-Strength: 0.75'

A 512×512 edit took about 10 s at 16 threads, and the edit pipeline is derived from the loaded one, so it needs no second load. 0.75 is the default. At 0.5 the picture keeps its composition and only starts to follow the prompt, and at 1.0 you get a new picture. A variation has no prompt to hold the subject: at 0.75 our test lighthouse kept its sunset and its coast and came back as a castle. Send a lower strength if you want more of the same thing.

Without a mask. There is no inpainting pipeline for an LCM in diffusers, so an edit that sends a mask is a 400 naming the recipe. It is refused before the model loads, not a minute later. sd15 inpaints on a CPU, at 30 steps.

It is billed 4 steps at any strength. An SD pipeline skips to int(steps × strength) and is billed for that. An LCM uses strength to pick where in the schedule to start, then runs all 4 steps. Billing it the SD way would have charged 2 at 0.5 and shown a progress bar reading 3 of 2. The worker now counts the steps a pipeline actually runs. If that differs from what was billed, the node logs STEP COUNT MISMATCH with the pipeline's name.

A negative prompt that would be ignored is refused (v3.56)

A negative prompt only does anything through classifier-free guidance, and a pipeline that is not running guidance drops it without a word. Until v3.56 that meant a 200 and a picture of exactly what you asked it to leave out. Now it is a 400 that says why, before the model loads, in two cases:

• The model has none. lcm-dreamshaper and flux-schnell (guidance-distilled; its pipeline only uses one under true_cfg_scale, which the recipe does not set). Recipes say so with "negativePrompt": false.
• This request's guidance is 1 or less. Every other pipeline in the catalogue skips the unconditional pass at guidance <= 1 (read from the pinned diffusers==0.36.0), so sdxl-turbo at its default 0.0 ignored every negative prompt it was sent. Raise X-InferHub-Image-Guidance above 1, or drop the negative prompt.

The worker's retry for a pipeline that rejects an argument also stopped dropping negative_prompt along with the progress callback. The callback may go; the negative prompt is refused instead.

Quantization is a property of the model, not of the request

"quantization": "none" | "int8" | "nf4", through diffusers' native bitsandbytes integration, applied to the components the recipe names. For qwen-image that has to include the text encoder: 8.3B left at bf16 is the difference between fitting on a 24 GB card and not.

It is a recipe field rather than a header because it changes what the model is. Two requests to qwen-image that quantized differently would produce different images from the same seed, and a per-request knob would make reproducibility a function of a header nobody logged. Want both? Ship two recipes with two ids — which is honest, and is also how you will describe it to your users.

One mechanism, deliberately. GGUF, Nunchaku and TensorRT are each faster on some model on some card, and each is a second thing to reason about when a picture comes out worse than expected.

A licence you have not read is a model this node will not start

Two of the seven are not permissively licensed. They are loaded, logged by name and not started until their licence id is in Tools:Image:AcceptedLicenses:

[diffusion] not offering 'sdxl-turbo': its licence is 'sai-nc-community', which is not permissive
            and is not in Tools:Image:AcceptedLicenses. Read it at https://huggingface.co/…/LICENSE.md
            and, if you accept it, add "sai-nc-community" to that list.

This is a fourth opt-in, and it is not redundant with the other three: Tools:Enabled consents to the feature, Tools:Allowed consents to these tools, Tools:AllowModelDownload consents to reaching the internet, and none of them says "and I accept the Stability AI Non-Commercial Research Community License". It is a list rather than a boolean for the same reason Tools:Allowed is: sd35-medium is free for most people who will run it and sdxl-turbo is not usable commercially at all, so one flag would let somebody who read one licence enable both.

A recipe that says nothing about its licence is treated as not permissive — a recipe that forgot to say is one nobody has read the licence of.

None of this is legal advice. It is a refusal to make that call on your behalf, silently.

The VRAM arithmetic, written down

Node:Vram:BudgetMiB is a number you set; Node:Vram:ReserveMiB (2048) is what is held back for the inference backend and the display. Unset means no gate and v3.15's behaviour exactly.

A recipe that cannot fit is not declared, so the fleet never routes at it and nobody spends a request finding out. One that would fit but does not right now — something else is mid-job on the card — waits on the tool queue and then gets the same 503 + Retry-After as every other limit here. Never an out-of-memory error inside somebody's job, which is the failure this replaces.

Declared, not detected, and that is the decision this section turns on. A node cannot reliably measure the card it is on: under WSL2 — the most common GPU-with-Docker setup there is — there are no /dev/nvidia* device nodes, the host's nvidia-smi cannot see the VM's VRAM, and the only reliable signal that a GPU exists at all is that libcuda.so.1 loads. A node that guessed would guess wrong on the exact platform this project is developed on. A budget that is usually right is worse than one that is explicitly absent, because the first failure is an OOM at 2am rather than a startup message.

The worker reports what it measures and the node logs a disagreement; it never overrides you.

A card that is also your desktop GPU (v3.50+)

Everything above assumes the card belongs to the node. On a desktop — one GPU, and you want it back between jobs — turn on Node:OnDemand:

"Node": {
  "OnDemand": { "Enabled": true, "ReleaseAfterSeconds": 30, "SwitchWaitSeconds": 300 }
}

The local Ollama and every tool (speech, transcription, image, video) then take the card one at a time:

1. A request arrives for a service that does not hold the card. It waits until the holder's in-flight work finishes — nothing is ever evicted mid-job.
2. The holder is released: Ollama unloads the models this node loaded (keep_alive: 0) — never one another program on the same box loaded through the same Ollama — and a tool's worker processes are stopped — CUDA context and all, which the idle hint cannot give back.
3. The newcomer starts, loads its model and runs.
4. Once a service has been quiet for ReleaseAfterSeconds it is released even if nobody else asked, so the card is free for you. 0 releases the moment the last request ends.

Requests for the service already holding the card share it (two chats run side by side). Once somebody is waiting for the card, new requests for the holder queue behind them, so a steady stream of chat cannot starve a video job. A request that waits past SwitchWaitSeconds gets the usual 503 + Retry-After. What the node declares to the coordinator does not change — a stopped tool is started again by its next request.

The price is cold starts. Every switch pays a model load: seconds for Whisper or Piper, a minute or more for FLUX or a large LLM. Alternating services request by request is slow by design; this mode is for a box that does one kind of job at a time and should be idle in between.

With several such boxes, the coordinator avoids the switch where it can (v3.51+). Each on-demand node reports who holds its card on every heartbeat, and again the moment it changes hands. When more than one node can serve a request, the router prefers, in order: a node whose card already holds the service the request needs, a node whose card is free, and only then one that would have to switch. It is a preference and never a refusal: if the only node holding the model is busy with something else, the request still goes there and waits, exactly as before. A node that does not run on demand counts as always warm, so a fleet without on-demand nodes routes exactly as it did. /api/status shows each node's onDemand state, the console shows gpu <holder> / gpu free / gpu switching, and inferhub_node_gpu_holder{node,holder} is on /metrics. The coordinator never tells a node to take or give back its card — that stays the node operator's decision. There is no new setting.

On a single card, the :all image is the easy way in (v3.52+). It holds everything :tools and :diffusion hold (Ollama, Whisper, Piper, the reranker and every diffusion recipe) in one node, and it ships with Node:OnDemand:Enabled=true. It is the only image that turns this on. Do not run :tools and :diffusion side by side on one card: that is two nodes, and neither can see what the other has loaded.

docker compose -f deploy/docker/compose.all.yml up -d

Switching models swaps weights; it does not restart anything

Loading FLUX is 40–90 seconds. A pool that restarted the process per recipe would pay the interpreter and the import of torch on top of that, on every alternation. So the worker frees the old pipeline, empties the cache, loads the new one, and reports the swap in the result's timing block — so a slow request has a visible reason rather than being the one somebody remembers as "it was slow that time".

Tools:Image:ResidentRecipes (default 1) allows more than one resident where the budget permits: a 48 GB card genuinely can hold SDXL and FLUX together and should not thrash. The default is 1 because the expensive default is the one nobody realises they chose.

After idleTimeoutSeconds the node sends the worker an idle hint and the worker frees its VRAM and stays alive. What to free is the worker's business — the node knows nothing about torch, and a node-side unload would be the node reaching into a tool's internals.

Weights arrive by an explicit pull, or in the background

FLUX is ~24 GB on the wire and Qwen-Image is larger. Since v3.16 you pull them deliberately, on the model-command channel the fleet has had since v2.8:

POST   /api/admin/nodes/{nodeId}/tools/diffusion/models/flux-schnell/pull
DELETE /api/admin/nodes/{nodeId}/tools/diffusion/models/flux-schnell

Progress relays on the existing /api/admin/stream as model-progress events, so the console gets a progress bar for free — along with the coalescing (a second pull of the same model rides the first) and the property that a hub restart forgets in-flight commands like everything else.

The progress carries no percentage, deliberately: Hugging Face gives no download callback, and a denominator we would have to guess is a number a dashboard would happily plot. What it reports is a fact — how many mebibytes have landed.

A generation request for a recipe whose weights are absent is a failed job naming both that command and the huggingface-cli one. Never a forty-minute wait.

On a fresh volume the node also fetches in the background and starts with capabilities: [], filling in as each model lands:

[diffusion] offering recipes: none yet (fetching: sd15, sdxl)
[diffusion] fetching weights for 'sd15' from stable-diffusion-v1-5/…@451f4fe16113 (variant=fp16)
[diffusion] 'sd15' is ready; offering recipes: sd15
[diffusion] 'sdxl' is ready; offering recipes: sd15, sdxl

No request ever waits on a download. A recipe is declared only once its weights are proven loadable, so the fleet never routes at a model that is not there — and the node tells its coordinator the moment one becomes available, without a restart.

That is v3.14.1, and v3.14.0 got it wrong: it fetched inside the request that first named the model, so the first sdxl call on a fresh volume spent the whole 900-second request budget downloading and returned a 502. If you want the weights before the container ever runs:

huggingface-cli download stabilityai/stable-diffusion-xl-base-1.0 \
  --revision 462165984030d82259a11f4367a4eed129e94a7b \
  --include "*.fp16.safetensors" "*.json" "*.txt" "*/*"

With Tools:AllowModelDownload=false that command is exactly what the log tells you to run.

Sizes are a list, not a range

SDXL was trained on fixed aspect buckets. A size outside them does not fail — it produces duplicated limbs and doubled horizons, which reads as "this model is bad" rather than "you asked for 1000×1000". So a size the recipe does not have is a 400 naming the ones it does:

Qwen-Image publishes a 4:3 bucket at 1472×1140, and 1140 is not a multiple of 8 — the edge refuses it before the request reaches a node. Rather than ship a size that 400s or quietly round it to a neighbour, qwen-image offers the three buckets that are expressible.

Steps, guidance and seed

InferHub extensions, as headers — additive by construction, so they cannot collide with whatever OpenAI adds next. An unknown value is a 400, never a silent fallback.

negative_prompt is a body field, deliberately: it is your own words, and a header is the one part of a request every proxy in the path writes into a log.

Every returned image carries the seed that produced it, so the one of four you liked is reproducible without re-rolling the other three.

360° panoramas (v3.17)

qwen-360 is ProGamerGov/qwen-360-diffusion — a rank-128 LoRA, MIT licensed, over Qwen-Image's 20B MMDiT — and it produces equirectangular panoramas whose left edge continues into their right edge.

Raise the step count. Measured in v3.28: at the recipe's default of 25 steps a 2048×1024 panorama comes out visibly under-denoised — a fine mottled speckle over stone, sky and ground alike. At 50 (X-InferHub-Image-Steps: 50, the recipe's maxSteps) with the same seed it is gone: 181 s instead of 107 s, and twice the megapixel-steps on the meter. seam_delta moves by 0.0016 between those two images, so nothing in the pipeline will tell you — this is a judgement the metric cannot make for you. Note that steps, guidance and seed travel as X-InferHub-Image-* headers; only seed is also read from the body.

curl http://localhost:5080/api/images/jobs \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qwen-360","prompt":"a Bulgarian mountain monastery courtyard at golden hour, photograph","size":"2048x1024"}'

The response, once it succeeds, carries the projection on every image:

{
  "created": 1785000000,
  "data": [{ "b64_json": "iVBORw0…", "size": "2048x1024", "seed": 42,
             "projection": "equirectangular", "seam_delta": 0.014, "revised_prompt": null }],
  "prompt_augmented": true,
  "trigger": "360 degree panorama with equirectangular projection"
}

A 2:1 aspect is not a suggestion. 360° of longitude over 180° of latitude is exactly two to one, and a render at any other ratio does not fail — it produces a panorama that looks perfectly fine flat and wraps wrongly in a viewer, which somebody discovers three days later wearing a headset. So a non-2:1 size for this recipe is a 400 that says why, not only which sizes exist.

The trigger phrase is appended, not demanded and not silently inserted. The model wants "equirectangular" or "360 panorama" in the prompt. Refusing a prompt without it would make the first request everybody sends a 400; rewriting one silently would be the one thing nothing in InferHub does. So it is appended when absent and the response says so — prompt_augmented, plus the phrase. Turn it off with "autoTrigger": false in the recipe; the flag is reported either way. The trigger is a recipe constant, so unlike your prompt it may appear in a log — which matters, because "why does this not look like a panorama" is almost always "the trigger did not apply".

The seam is measured on every panorama, and repaired only if you ask. seam_delta is the mean absolute difference between the first and last columns — the pair that becomes adjacent once the image is wrapped — normalised to 0–1. Over Tools:Image:SeamWarnThreshold (default 0.08) the result carries a "warnings": ["seam"] entry. It is a warning and never a failure: a slightly visible seam is your own problem and your own aesthetic judgement.

Since v3.23 you can close it, through two gates. The operator sets Tools:Image:SeamRepair on the node — off by default, then blend, diffuse or any — and the request picks within that:

curl http://localhost:5080/v1/images/generations \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -H 'X-InferHub-Image-Seam-Repair: blend' \
  -d '{"model":"qwen-360","prompt":"a lighthouse at dusk","size":"2048x1024"}'

# → 200  "seam_delta": 0.011, "seam_delta_before": 0.134, "seam_repair": "blend"

Nothing repairs by default, and no threshold ever triggers one. The threshold decides whether to warn; a number that decides to spend your GPU is the tool overriding you with a helpful expression on. Send no header and the response is what v3.22 sent, byte for byte.

• blend is a wrapped feather across the join: numpy on the array the VAE already produced — milliseconds, no VRAM, no steps and nothing on your bill. It closes a tonal discontinuity and not a structural one: a seam through a doorway comes back with no visible step in brightness and the doorway still not lining up. That is what the other one is for.
• diffuse rolls the join into the middle of the picture and inpaints a band of it — upstream's fix_seam, over the same resident weights. It costs int(steps × 0.4) more steps, metered in megapixel_steps and included in total_steps from the first progress frame, so the bar never restarts.

diffuse does not imply blend: these name mechanisms rather than tiers, so an operator who thinks a feathered band is worse than an honest seam can permit only the real repair. any is how you say both. A repair that does not lower the number is discarded and says so — you get the original image, the mechanism, and two equal numbers, because a pass that quietly made your panorama worse is the one outcome nobody would ever go looking for. seam_delta is always the image you were handed; seam_delta_before is what it measured first. On the content route the same three facts arrive as X-InferHub-Image-Seam-Repair, -Seam-Delta and -Seam-Delta-Before.

Since v3.53 a panorama can come back as a cubemap, which is what a game engine, three.js' CubeTextureLoader or a KTX/DDS cube texture wants instead of a 2:1 sphere:

curl http://localhost:5080/v1/images/generations   -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json'   -H 'X-InferHub-Image-Reproject: cubemap'   -d '{"model":"qwen-360","prompt":"a lighthouse at dusk","size":"2048x1024"}'

# → 200  "projection": "cubemap", "size": "2048x1024" — and the PNG is 3072x512

• One PNG, six square faces in a row, left to right +X −X +Y −Y +Z −Z — the OpenGL cube-map order, which three.js, Babylon, KTX and DDS share — each width / 4 on a side, oriented as the OpenGL cube-map table defines it. The panorama's centre column faces +X (three.js' own equirectangular origin), so its join lands on the vertical centre line of −X.
• size stays the render's. The steps ran on the 2048×1024 render and that is what you are billed for. "projection": "cubemap" is what tells you the bytes are a 6·(w/4) × (w/4) strip.
• The seam comes first. seam_delta and a repair describe the panorama the faces were cut from, so X-InferHub-Image-Seam-Repair: blend plus cubemap closes the join and then cuts the cube.
• It is about a second of CPU at the largest size, no VRAM and no steps, so there is no operator key: nothing is spent that needs a ceiling. A flat recipe is refused, since a flat picture is not a sphere, and an unknown value is a 400 naming cubemap. If the cut itself fails, you get the panorama back declared equirectangular with a reproject warning, so a two-minute render is never thrown away and the response still says what the bytes are.

Where the projection turns up: in the response body per image, on the job document (GET /api/images/jobs/{id}), and as X-InferHub-Image-Projection on the content route — which is the one request that has no JSON to read it from. A flat recipe reports "flat" rather than omitting the field, because an absent projection is indistinguishable from a node too old to have one.

A viewer ships with it, at /console.html → 360° viewer: paste a job id, and it picks its renderer from the declared projection rather than from the aspect ratio — which is what everything else does, and is wrong for every 2:1 landscape photo. It is wwwroot/pano.js, ~330 lines of hand-written WebGL, no npm, no CDN script, no three.js.

qwen-image and qwen-360 are two model ids over one base. Not one model with a scale parameter: the router keys on (capability, model) and a client asking for qwen-image must never receive a panorama. The worker keeps the base resident and swaps only the LoRA when you alternate between them, which is seconds rather than the 40–90 s a 20B reload costs — an optimisation, invisible to the contract, and reported in the result's timing block.

You need a card, and this says so rather than being quietly slow

Tools:Image:RequireGpu defaults to true: with no reachable CUDA device the worker refuses to start and names the key to unset. A tool that loads happily on a CPU and then serves four-minute requests is a node the fleet keeps routing to, and every caller pays for the discovery.

Unset it and only recipes marked cpuViable are offered — sd15 at 512×512, and not sdxl. Tools:Image:AllowSlowCpu=true offers the rest anyway: your hardware, your call. There is no "CPU ✅" anywhere in this README for the feature as a whole, because that tick would be true of one recipe and a lie about the other.

The worker's first log line names the device it picked. Four gigabytes of CUDA and a silent CPU fallback is an afternoon of blaming the model.

No URL, no gallery, no prompt in a log

• response_format=url is a 400 naming b64_json. Serving a URL means the hub keeps the bytes, and keeping the bytes means an image store, a retention window and a question about whose pictures those are that we have not agreed to answer.
• A prompt is content. Nothing logs one, at any level, on either host. The line for a generation carries the model, the image count, the megapixel-steps and the outcome.
• The node writes images into a per-request scratch directory that is deleted in a finally, always — after success and after failure.
• There is no bundled safety classifier, and that is a decision: diffusers' checker returns a black image, which is indistinguishable from a broken VAE, a bad seed or an OOM, so the operator gets a bug report instead of a policy signal. This box generates what you ask it to generate; the policy is yours.

ImagePrivacyTests runs a generation through a real mesh with a capturing logger and fails if the prompt appears anywhere in the logs or the ledger.

Batches are bounded by bytes, not by count

n is capped by Images:MaxBatch (4) and by an upper-bound byte estimate checked before a step runs, with the refusal naming the largest n that fits at that size. The budget is clamped by Tools:MaxAttachmentBytes, because that cap is what sizes the mesh's SignalR message limit — and exceeding a SignalR limit tears the node's connection down rather than failing the message. Raising one without the other gets you no change, deliberately.

A batch runs on one node: fanning it across the fleet would mean four different seeds' worth of scheduling for a request you think is atomic, and a partial failure would have no honest status.

Usage and quotas

Metered in megapixel-steps — width × height × steps / 1e6, from what the worker actually produced. Not "images": a 512² render at 4 steps and a 2048×1024 one at 30 steps are both one image, and the second is 47× the work.

"Auth": { "Clients": [ { "Id": "acme", "Key": "…",
  "Limits": { "MegapixelStepsPerDay": 5000 } } ] }

Over it is a 402 with Retry-After pointing at UTC midnight, the same shape as every other budget.

Editing a picture (v3.18+)

Inpainting, image-to-image and variations — on OpenAI's edits API, multipart and all.

curl http://localhost:5080/v1/images/edits -H "Authorization: Bearer $KEY" \
  -F model=sdxl -F image=@room.png -F mask=@mask.png \
  -F prompt="a tall window with morning light" \
  -H 'X-InferHub-Image-Strength: 0.75' \
  | jq -r '.data[0].b64_json' | base64 -d > edited.png

Three shapes, and the difference between them is what you send:

The mask convention everybody gets backwards

OpenAI's edits API treats a fully transparent pixel as the area to edit. diffusers takes a mask where white is the area to inpaint. These are opposite.

   your mask (OpenAI)              what the pipeline wants
   ┌──────────────────┐            ┌──────────────────┐
   │▓▓▓▓▓▓░░░░░▓▓▓▓▓▓▓│            │      █████       │
   │▓▓▓▓▓▓░░░░░▓▓▓▓▓▓▓│   ──────▶  │      █████       │
   │▓▓▓▓▓▓░░░░░▓▓▓▓▓▓▓│            │      █████       │
   └──────────────────┘            └──────────────────┘
    ▓ opaque = keep                  █ white = inpaint
    ░ transparent = EDIT THIS        (blank = keep)

Get it backwards and nothing errors — it edits everything except what you selected, which reads as a broken model rather than a backwards mask. So the conversion is explicit, it happens in the worker (the only thing in the path that may open a PNG), and there are two refusals rather than a guess:

• A mask with no alpha channel is a 400. Under OpenAI's convention a fully opaque image selects nothing, which nobody has ever intended; reading it as "edit everything" would be a silent substitution of the most destructive possible interpretation.
• A mask whose size differs from the image is a 400 naming both. A mask names which pixels, so it is never rescaled — the edit would land next to what you selected.

Already have a white-is-edit mask? Say so:

X-InferHub-Mask-Convention: luminance      # openai (default) | luminance

An unknown value is a 400 that names both and says which is which, because two words whose difference is invisible until you look at the picture are not a helpful list.

Strength is a header, and it is the whole knob

OpenAI's edits API has no strength, and image-to-image without one is meaningless.

X-InferHub-Image-Strength: 0.75      # 0 keeps your picture, 1 ignores it

Absent, the recipe's defaults.strength applies (0.75 for every editable recipe). Out of range is a 400. It is a header rather than a body field for the same reason steps and guidance are: additive by construction, so it cannot collide with whatever OpenAI adds next.

What gets metered is the steps that actually ran. diffusers enters the schedule at int(steps × strength), so 30 steps at 0.6 denoises for 18 — and 18 is what the ledger gets. Billing the asked-for 30 would charge for work nobody did. lcm-dreamshaper is the exception, and it runs more, not less: its scheduler uses strength to pick where to start and still runs all 4 steps, so 4 is what it reports at any strength (v3.57).

Not every model can edit

Editing is its own capability, image-edit, declared per recipe from its operations field:

FLUX.1-schnell has no official inpainting pipeline; SDXL does. So an edit against a generate-only recipe is a 503 that names the ones that can:

{ "error": { "code": "capability_unavailable", "message":
  "no node currently provides 'image-edit' for model 'flux-schnell'. Models on this fleet that do: sd15, sdxl" } }

That is a fleet-state answer rather than an authorization one, which is the distinction v3.8 wrote down. A second capability kind rather than a per-model operation list, because the router filters on (capability, model) and nothing else — and teaching it to read a nested operation set would mean teaching the affinity, the queue and the saturation logic the same thing.

Editing is a job like any other

Same queue, same per-step progress, same cooperative cancel, same five-minute in-memory retention. The async route takes multipart too, and names its operation:

curl http://localhost:5080/api/images/jobs -H "Authorization: Bearer $KEY" \
  -F operation=edit -F model=sdxl -F image=@room.png -F mask=@mask.png \
  -F prompt="a tall window with morning light"

JSON generates; multipart edits. A multipart submission that names no operation is a 400 naming both — this is InferHub's own contract, where ceremony is cheaper than a silent substitution.

What comes in is content too

Images:MaxRequestBytes (25 MB) caps the picture and the mask together, refused with a 413 at the edge before anything is buffered onward; each part is additionally capped by Tools:MaxAttachmentBytes. Nothing is retained: the bytes are held for the dispatch, written into a per-request scratch directory the node deletes in a finally, and dropped. The filename you chose never leaves the edge — the parts travel as image and mask, because what you called a file on your disk is metadata about your day.

Not in v3.18, and said out loud: ControlNet, IP-Adapter and reference-image conditioning (each is a per-base-model zoo of auxiliary weights with its own preprocessors, and every preprocessor is image processing); an outpainting helper (it is inpainting with a canvas you prepare, and preparing it on the hub would mean decoding a pixel); multi-image edit chains; and FLUX inpainting, which has no official pipeline to wrap.

A job that takes two minutes (v3.15+)

An SDXL render at 50 steps is not a request, it is a job. Before v3.15 you sent one and held a connection open, and the only two things you could learn were "it worked" and "something timed out" — usually from a proxy in the path, with a status nobody could act on.

# submit
ID=$(curl -sS http://localhost:5080/api/images/jobs -H "Authorization: Bearer $KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"sdxl","prompt":"a lighthouse in a storm","size":"1024x1024"}' | jq -r .id)

# watch  (SSE: queued → running(step 7/28) → succeeded)
curl -N http://localhost:5080/api/images/jobs/$ID/events -H "Authorization: Bearer $KEY"

# collect (read-once, in memory, expiring)
curl -sS http://localhost:5080/api/images/jobs/$ID/content/0 -H "Authorization: Bearer $KEY" -o out.png

# or change your mind
curl -X DELETE http://localhost:5080/api/images/jobs/$ID -H "Authorization: Bearer $KEY"

A submitted job answers 202 with a place in line, not a wait-then-503: you already accepted asynchrony, so making you retry would be strictly worse than telling you where you are. The states are queued → running → succeeded | failed | cancelled | expired, with cancelling in between, and the transition table is data both hosts and the tests read from the same place.

There is no background: true flag, and that is on purpose

OpenAI has no asynchronous Images API to adopt, and where there is no dialect to adopt this project does not invent an OpenAI-shaped one — work with no existing shape travels as its own honest contract under /api. A background: true field on /v1/images/generations returning a non-OpenAI body would make one route answer two incompatible shapes depending on a flag, which every typed SDK gets wrong, and would turn "is this endpoint OpenAI-compatible?" into a question with a footnote.

/v1/images/generations is unchanged. Internally it became "submit a job and wait for it", so a synchronous call and an asynchronous one queue in the same line and are metered by the same code — the alternative, two paths to a GPU with two ideas of fairness, is how a fleet grows a fast lane nobody documented. Past Images:SyncMaxWaitSeconds (120) it answers 503 naming the job id and the async route; the job keeps running, because throwing away a minute of GPU because an HTTP client got bored is your decision, not the hub's.

Cancel does not kill the worker

A DELETE sends a cancel frame down to the worker, which honours it from its per-step callback and answers with an error coded cancelled. Then it is still alive, still holding its weights.

Killing it would be simpler and is wrong. A diffusion worker's weights took tens of seconds to load — and twelve to twenty gigabytes taking a minute or more once the catalogue grows — so killing it to abandon one job punishes the next caller for your change of mind, and the punishment gets worse with every model added. A worker that has not answered within Tools:CancelGraceSeconds (20) is by definition not cooperating, and that one is terminated and restarted.

Cancellation is best-effort and this says so. A job cancelled at step 27 of 28 may still succeed, and if it does you get your image. Discarding a finished result to honour a state name would be worse than telling you what actually happened.

Results live for five minutes, in memory by default

• Images:Jobs:RetentionSeconds (300) — a finished job's record and bytes are dropped this long after completion, read or not.
• Images:Jobs:MaxRetainedBytes (512 MB) — a global ceiling, LRU-evicting completed results and never an in-flight one, enforced on insert rather than on a timer. An evicted job reads as expired with a reason, so arriving late is a 410 that says what happened rather than a 404 that looks like a bug.
• Read-once by default — a delivered image is dropped immediately. Images:Jobs:KeepAfterRead exists for a console's benefit and is the setting that makes the hub briefly an image cache.
• Nothing touches disk by default. No temp file, no spill under memory pressure, no cache directory. Under pressure the answer is eviction and a 503 on submit, not a file. A restart forgets in-flight and completed jobs, exactly like every other counter on the hub.

A job can survive a restart, if you say so (v3.24)

Images:Jobs:Persistence=file writes a finished job's record and its bytes under Images:Jobs:DataDirectory, so a deploy no longer turns a job id from thirty seconds ago into a 404. It is off by default, and turning it on is answering a data-retention question rather than flipping a performance switch — which is why it was argued as the fourth exception to this project's "no persisted state" rule rather than added quietly. What that argument bought you:

• Durability does not extend retention. The window is applied on load: a hub that was down for an hour comes back and deletes everything past RetentionSeconds before it serves the first request. Restarting is never a way to keep a picture longer than you allowed.
• Read-once means read-once from the disk too. Delivery, eviction and expiry each unlink the file in the same operation that drops the bytes, so the API's answer and the directory's contents cannot disagree.
• An interrupted job is never resumed. It comes back failed with reason hub_restarted and a sentence saying so, because nothing durable holds your prompt — there is no field for one, and a re-dispatch would have needed it. Submitting again is your call, exactly as it is when a node disappears mid-job.
• There is no postgres here, deliberately. Image bytes are not row data, and half a gigabyte of PNGs in a bytea column is WAL amplification per render plus a database dump that now contains pictures.
• Under Cluster:Enabled this is per instance. A promoted standby does not hold the old primary's pictures. The hubs share a Postgres, not a directory.

The container images set Images__Jobs__DataDirectory=/data/images, so mounting a volume at /data is what makes it survive a docker run as well as a restart.

The queue is FIFO and it is not clever

A GPU running diffusion is a resource there is exactly one of, so the hub gives each capable node one image job at a time and takes the queue in order. Shortest-job-first would let a stream of four-step requests starve a fifty-step one indefinitely and the starvation would be invisible; fair-share needs a notion of tenant weight this project's client model does not have. Images:Jobs:MaxQueueDepth (32) bounds it, and a full queue is 503 + Retry-After — the same status and header as every other limit here, so your retry logic behaves identically whichever one it hit.

A node that disappears mid-job fails the job, and never retries it

An image job that died at step 22 produced no output, so it is technically retryable — and retrying it would silently double the GPU-minutes and the ledger units for one request. It fails with node_lost and you decide. A job still queued when its node goes away has spent nothing and is simply routed again.

Every route is yours only

A job id belonging to another client is a 404, byte-identical to one that does not exist — never a 403. On a surface whose ids are only knowable by having been issued one, the difference between "not yours" and "not there" is the isolation boundary.

Solo mode gets the same five routes

A standalone node serves /api/images/jobs itself, with the same bodies and the same statuses — one job at a time, because it is a box with a card in it. The deployment least likely to have a proxy that tolerates a two-minute request is the one somebody is running on a laptop.

Text to video (v3.25+)

A node with a card and the :diffusion image can render short clips, and the API is OpenAI's Videos API — not one of ours:

curl -s http://localhost:5080/v1/videos   -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json'   -d '{"model":"wan-t2v-1.3b","prompt":"a paper boat on a puddle, slow dolly in","seconds":5}'
# {"id":"video_8f3c…","object":"video","model":"wan-t2v-1.3b","status":"queued","progress":0,
#  "seconds":5,"size":"832x480","created_at":1755400000}

curl -s http://localhost:5080/v1/videos/video_8f3c… -H "Authorization: Bearer $KEY"
# …"status":"in_progress","progress":42…

curl -s http://localhost:5080/v1/videos/video_8f3c…/content -H "Authorization: Bearer $KEY" -o out.mp4

GET /v1/videos/{id} polls, /content fetches the bytes once, and DELETE cancels and drops it. That is the whole surface.

Measured in v3.28, on a 3090 Ti, and worth knowing before you first run one. A five-second wan-t2v-1.3b clip at 832×480 is ~143 s of cold model load plus ~340 s of generation — 378 s end to end, 81 frames, 982 KB of H.264. Two things follow. Set Dispatcher:Deadlines:video to 3600 (v3.54; before it, raise Dispatcher:TimeoutSeconds for everything): the default of 300 covers tool jobs too and gives up while the model is still loading. And the first pull is ~29 GB, because "1.3B" names the transformer only — the UMT5 text encoder is ~11B and every weight in that repository is fp32 with no fp16 variant.

Why this one is adopted and the image one was invented

v3.15 built /api/images/jobs because OpenAI has no asynchronous Images API to adopt, and this project's rule is to speak the dialect clients already speak and to invent only where there is none. Video has one, and it is asynchronous by construction — create, poll, fetch, delete — so client.videos.create(...) in any OpenAI SDK works against a hub that can serve it.

Two of that dialect's routes are 501s that say why rather than 404s that read like an old server. GET /v1/videos would enumerate your jobs, and this coordinator holds no such index — the id it handed you is the capability to fetch the bytes. POST /v1/videos/{id}/remix needs the request kept after the job ends, and nothing durable here holds a prompt.

It is the same job model, so it is the same everything else

The queue, the per-step progress, the cooperative cancel that leaves the worker holding its weights, the five-minute read-once retention and v3.24's optional durability are the code the image jobs already run. Images:Jobs:* configures both, and Images:MaxResponseBytes bounds both — one wire, one ceiling, clamped by Tools:MaxAttachmentBytes as always.

A video id will not open an image route and an image id will not open a video route: both are the same 404 an unknown id earns.

Three models since v3.26, and four things worth knowing about all of them

All three are Apache-2.0, so none of them needs a licence decision. wan-t2v-14b-720p does not fit a 24 GB card, and that is the VRAM gate doing its job rather than an oversight: a node with 24 GB never declares it, so the hub never routes to it and nobody finds the ceiling four minutes into a render. A recipe's figure is sized at the largest (size, seconds) pair it offers, because the gate is handed one number before any caller exists.

cogvideox-2b is why fps is a required field. It runs at 8 where Wan runs at 16, and a worker that fell back to 16 would encode its 49 frames at twice their rate — not an error, a clip that plays at double speed. 49 frames at 8 fps is 6.125 s, so seconds: 6 is the label and 6.125 is what comes back.

• "1.3B" names the transformer only. The text encoder beside it is UMT5-XXL at ~11B, every weight in the repo is stored fp32 with no fp16 variant, and the first pull is ~29 GB. The recipe declares 15 500 MiB of VRAM for that reason, not for 1.3B of it.
• Sizes divide by 16, not by 8. 840x480 is a perfectly good image size and an invalid video one; the refusal says so at the edge rather than four minutes into a job.
• seconds is a label, and the response reports the measurement. A latent video pipeline puts frames on a 4k+1 grid, so seconds: 5 means 81 frames and 81 frames at 16 fps is 5.06 s — which is what comes back. A duration the model does not offer is refused naming the list, never rounded to the nearest one.
• fps is not yours to set — and since v3.26 the recipe must state it. It is the rate the model was trained at; re-timing the frames at encode changes how fast the world moves. A recipe that does not declare it is skipped and logged by name rather than assumed to be 16.

Usage, and the two units

A video meters both: megapixel_steps — width × height × frames × steps / 1e6, because a video transformer denoises the whole latent stack on every step — and video_seconds. The first is the card's real cost and spends the same MegapixelStepsPerDay budget an image does, because it is the same card: a 5-second 832×480 clip at 30 steps is about 970 megapixel-steps against an SDXL image's 31. The second is the number a human asks about, and neither can be derived from the other. inferhub_video_seconds_total{kind,model} is on /metrics, and emits nothing at all on a fleet that has never rendered one.

Since v3.27 both units are also gates. VideoSecondsPerDay is the budget in the unit a person sizes; MegapixelStepsPerDay remains the one that describes the card. A submission is refused if either is spent, with a 402 naming which — before v3.27 only the megapixel-step budget was checked, so a fleet that meant "an hour of video a day" had no way to say it.

Watching one from the console

/console has a Video panel: submit a prompt, watch the row (queue position, step n of m, elapsed, which node), cancel, and play the clip in the page. It speaks /v1/videos — the same routes an SDK calls — for everything except listing, which that dialect refuses on purpose; the listing is GET /api/videos/jobs and is client-scoped, so the panel holds a client key rather than the admin key. A fetched clip is consumed: it lives in the browser tab and the hub has dropped its copy.

Underneath it, Video recipes on the fleet answers "why can I not use that model" for clips the way the Images panel does for pictures — the licence, the card, a profile, or weights that are not there yet. wan-t2v-14b-720p showing over-budget on a 24 GB box is the ceiling working, not a fault, and the row says so.

What the video track does not do yet

No image-to-video — a second capability and a second input path, and it is named rather than forgotten. No caller-chosen fps. No audio: Wan2.1 T2V produces none, and a silent track added to look complete is a lie in a container. No 480p entry for the 14B — the same weights at a second geometry means two recipe ids over one loaded pipeline, which the residency map would count twice against one card; that is a phase, not a JSON file. No listing on /v1/videos — an id is the capability to fetch the bytes, so the dialect's enumeration stays a 501 that says why, and the console's listing is a client-scoped route of our own. And no video has been watched: every claim above about these models comes from their own configs, their model cards and the pinned diffusers wheel, not from a card.

Configure the fleet, not the boxes (v3.11+)

Twenty nodes means twenty appsettings.json files and twenty restarts. A node profile is the coordinator saying what a node should be doing — and the node deciding whether it may:

curl -X PUT http://localhost:5080/api/admin/profiles/gpu-boxes \
  -H "Authorization: Bearer $ADMIN" -H 'Content-Type: application/json' -d '{
    "selector": { "labels": { "tier": "gpu" } },
    "capabilities": { "embed": false },
    "tools": { "whisper": true, "piper": false },
    "models": { "ensure": ["llama3.2"], "remove": [] },
    "maxConcurrency": 4
  }'

Every node whose labels match applies what it can, and reports back what it would not do and why. GET /api/admin/nodes/{id}/profile is the answer to "I wrote that and the box still does what it did before".

The hub can only ever narrow a node, and the check runs on the node

This is the design decision the feature exists around, and most systems in this space get it the other way round. A profile can switch a capability off; it cannot re-open one the box's own Node:Capabilities:Disabled closed. It can stop a tool; it cannot introduce one — a tool id that is not already in that node's Tools:Allowed is refused by name, and there is no field anywhere in a profile for a command, a path or an interpreter. It can lower MaxConcurrency; raising it is refused, because that number is a statement about hardware you own and the coordinator does not.

The clamp that enforces all of this runs on the node, not on the hub. A clamp on the hub is a clamp an attacker skips by not being the hub: the point is that a compromised or misconfigured coordinator cannot turn a fleet of GPU boxes into fleet-wide remote code execution. What a profile can do is bounded by what you already put in the file on the machine.

Desired state, so a rebooted node fixes itself

A profile is not a command. A node asks for its profile every time it registers, so a box that was being rebuilt when you wrote one converges on the way back in, with no operator action and nothing for the hub to remember. Re-applying the same revision changes nothing and says so — which is why that reconnect is safe to run unconditionally instead of re-pulling forty gigabytes of weights.

Refusals are per item: a profile asking for one impossible thing and four possible ones applies the four and reports the one. A profile is never a startup dependency and never restarts a node — switching a tool off stops its workers in place, and switching it back on starts them again.

Selectors, conflicts and persistence

Selectors are exact: a nodeId, or a set of labels of which every pair must match. No globs, no expression language — a pattern dialect pointed at a security boundary is how somebody's node ends up matched by a rule that reads correct. A selector that names nothing matches nothing, and is refused with a 400 rather than quietly applying to every box you own.

If two profiles match one node, neither is sent. The node keeps what it last applied and /api/status and the console show it as conflict until you fix the selectors — silent precedence is how a node ends up in a state no single document explains.

Profiles are in memory by default and a coordinator restart forgets them; set Fleet:Profiles:Persistence to file or postgres to keep them (postgres is what an HA pair wants, so both hubs read one fleet configuration). Losing them is survivable by design: every node falls back to its own configuration, which is never a wrong answer and never a capability nobody granted.

A corpus on every node (v3.12+)

Retrieval used to be the hub's, or a standalone node's, and never both. From v3.12 the coordinator can turn retrieval on for a node, choose its vector engine, and have the node bring it up — without editing a file on the box and without restarting it:

curl -X PUT http://localhost:5080/api/admin/profiles/edge-boxes \
  -H "Authorization: Bearer $ADMIN" -H 'Content-Type: application/json' -d '{
    "selector": { "labels": { "site": "sofia" } },
    "retrieval": {
      "enabled": true,
      "provider": "qdrant",
      "url": "http://qdrant.sofia.internal:6333",
      "credentialRef": "sofia-qdrant",
      "collections": ["site-sofia-docs"],
      "embeddingModel": "nomic-embed-text"
    }
  }'

The node starts the corpus while it keeps answering chat. Switch it off and the retrieval routes go back to 501 with the sentence they have always had, the in-flight retrievals drain rather than fault, and the corpus is still there when you switch it back on.

One authority per collection name, and the hub knows who it is

v3.6 refused to let a meshed node hold its own corpus, because a node with hub-derived replicas and an authority under the same names is a collection with two truths and a replication pass waiting to overwrite one of them. That rule is not relaxed — a node that sets LocalApi:Retrieval:Enabled by hand while meshed still refuses to boot.

What changed is that the hub can now be the one who assigns the corpus, and can therefore be the one who knows:

• The hub records an owner per collection: hub, or node:{id}.
• A hub-side create of a node-owned name is a 409 naming the owner — not a mystery conflict.
• Replication and healing never target a node-owned collection. There is nothing hub-side to derive replicas from, and pushing an empty snapshot at the owner would be the hub deleting somebody's corpus while reporting success.

The client API does not move

A client posts to /api/collections/{c}/documents and searches at /api/collections/{c}/search exactly as before, whether the chunks are in the hub's store or on a box in another city. The hub sees a node-owned collection and dispatches the work to its owner over the connection the node already opened — so client scoping, quotas and the response shapes are all the ones you already had.

Two things are said out loud rather than left to be discovered:

• PDF is a 415 on a node-owned collection. The PDF extractor ships with the coordinator only, and a hub that extracted the text and shipped chunks would be a second ingestion path with different chunking and a different failure mode. Convert to text or Markdown, or use a hub-owned collection.
• An owner that is not connected is a 503 naming the node, never a quiet answer from the hub's own store. A confident answer from the wrong corpus is the failure nobody notices.

Federated retrieval across corpora (v3.40+)

A single query can now fan out to several collections at once — hub-owned and node-owned, mixed — and come back as one ranked list:

curl localhost:5080/api/retrieve/federated -H 'Content-Type: application/json' -d '{
  "collections": ["site-sofia-docs", "release-notes"],
  "query": "how does node ownership work",
  "k": 5
}'

Each name runs through the exact single-collection path /search already uses — nothing about how a collection is searched changes — in parallel, bounded by a per-collection maxWaitMs (4s default). The ranked lists are combined by Reciprocal Rank Fusion, keyed by (collection, id) so two unrelated corpora's identically-named chunk ids are never merged into one. The response names every source's outcome:

{
  "matches": [{ "collection": "site-sofia-docs", "id": "c1", "score": 0.031, "payload": "…" }],
  "sources": [
    { "collection": "site-sofia-docs", "status": "ok", "matches": 3, "elapsedMs": 42 },
    { "collection": "release-notes", "status": "timeout", "matches": 0, "elapsedMs": 4000 }
  ]
}

A collection outside the caller's scope or that genuinely doesn't exist reports not_found — the two are indistinguishable, exactly as a single-collection request already keeps them (phase-31 D3). An owner that is offline reports unavailable rather than being silently dropped or answered from the wrong corpus. A slow source times out without failing the other collections' answers. At most 16 collections per call.

A node-owned collection can survive its owning node (v3.42+)

Until now, a node-owned collection lived only on that node's own disk — if the box was gone for good, so was the corpus. An admin can now assign a standby:

curl -X POST localhost:5080/api/admin/collections/site-sofia-docs/standby/gpu-box-2 \
  -H 'Authorization: Bearer <admin-key>'

While the primary is healthy, the hub relays its snapshot and every subsequent write to the standby — without ever holding a copy itself, the same "this hub deliberately holds no copy of it" guarantee a node-owned collection always had. If the primary is confirmed gone (not a reconnect blip — a configurable grace period, off by default via CorpusFailover:Enabled), the standby is promoted: it becomes the collection's new owner through the exact same profile mechanism an admin's own assignment uses. There is no automatic failback — a primary that comes back afterward holds a stray copy an operator resolves by hand.

Two things worth knowing: this covers the local provider only for now (qdrant/postgres-owned collections have no standby path yet), and it is off on both ends by default — a node forwards nothing unless LocalApi:Retrieval:ReplicateOwnedCollections is set, and the hub promotes nothing unless CorpusFailover:Enabled is set.

The engine, the secret and the disk stay the operator's

A profile names provider and credentialRef. It cannot name a secret: credentialRef is a name, resolved on the node against LocalApi:Retrieval:Credentials:{ref}, and a name the box does not have is a refusal naming the key — never a fall back to an unauthenticated connection to your Qdrant. There is no field anywhere in a profile for a data directory, either.

A node runs local (its own disk), qdrant or, since v3.38, postgres — the same PostgresVectorStore the coordinator uses, shared via a project the node references directly. The one asymmetry with Qdrant: postgres has no credentialRef path, because a connection string already carries its own password and has nowhere clean to graft a hub-resolved secret onto — set LocalApi:Retrieval:Postgres:ConnectionString on the node itself.

A start that fails — unreachable engine, unresolvable credential, wrong dimension — leaves the node with no corpus and a reported refusal, visible on /api/status and in the console, while the node goes on serving chat.

One box, one container: chat, RAG and speech (v3.13+)

Six releases add up to one story, and the sections above tell it in pieces. Here it is once, top to bottom. A coordinator on a small always-on host, one GPU box, and everything else configured from the hub — no file edited on the GPU machine after the first docker run, no restart, and inference never stops while you do it.

1. The hub. Anywhere; no GPU.

docker run -d --name inferhub-hub \
  -e Auth__AdminApiKeys__0="$ADMIN_KEY" \
  -e Auth__ApiKeys__0="$CLIENT_KEY" \
  -e Auth__NodeEnrollmentSecret="$ENROLL" \
  -v inferhub-hub:/data -p 5080:8080 \
  ghcr.io/dev-art-solutions/inferhub-coordinator

2. The GPU box. One container. It carries Ollama and the speech workers, dials out to the hub, and needs no inbound rule of its own. The labels are what a profile will select on in step 4.

docker run -d --name inferhub-node --gpus all \
  -e Coordinator__Url=http://hub.example:5080 \
  -e Coordinator__EnrollmentSecret="$ENROLL" \
  -e Node__Labels__role=gpu \
  -e Tools__Enabled=true -e Tools__Allowed__0=whisper -e Tools__Allowed__1=piper \
  -v inferhub-node:/data \
  ghcr.io/dev-art-solutions/inferhub-node:tools

docker exec inferhub-node ollama pull llama3.2

Tools:Allowed is the operator's grant and the ceiling the hub can never raise — step 4 can switch those two tools off, and can never introduce a third.

3. Chat, already. Point anything that speaks Ollama or OpenAI at the hub.

curl http://hub.example:5080/v1/chat/completions \
  -H "Authorization: Bearer $CLIENT_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"llama3.2","messages":[{"role":"user","content":"Hello!"}]}'

4. Give the box a corpus — from the hub, with no file edited on the box.

curl -X PUT http://hub.example:5080/api/admin/profiles/gpu-boxes \
  -H "Authorization: Bearer $ADMIN_KEY" -H 'Content-Type: application/json' \
  -d '{
        "name": "gpu-boxes",
        "selector": { "labels": { "role": "gpu" } },
        "maxConcurrency": 2,
        "retrieval": {
          "enabled": true, "provider": "local",
          "collections": ["handbook"], "embeddingModel": "all-minilm"
        }
      }'

The node pulls the profile, clamps it against its own configuration, brings the corpus up at runtime, and reports what it applied and what it refused. Ingest and search stay on the hub's own API — the hub dispatches them to the owner:

curl -X POST http://hub.example:5080/api/collections/handbook/documents \
  -H "Authorization: Bearer $CLIENT_KEY" -F file=@handbook.md

curl http://hub.example:5080/v1/chat/completions \
  -H "Authorization: Bearer $CLIENT_KEY" -H 'Content-Type: application/json' \
  -H 'X-InferHub-Retrieve: handbook' \
  -d '{"model":"llama3.2","messages":[{"role":"user","content":"What is our refund window?"}]}'

5. Speech, on the same box, with no extra deployment. The :tools image already carries the workers; routing is per (capability, model), so a node busy transcribing is still a candidate for chat.

curl -X POST http://hub.example:5080/v1/audio/transcriptions \
  -H "Authorization: Bearer $CLIENT_KEY" \
  -F file=@meeting.m4a -F model=whisper-small -F response_format=srt

6. Watch all of it on one page. http://hub.example:5080/console.html, admin key in the bar at the top.

Driving all of it from one page (v3.13+)

The console is plain HTML, CSS and JavaScript served by the coordinator — no build step, no bundler, no framework, and nothing to install. It reads /api/status and /api/admin/* and adds no API of its own.

Desired beside effective, always. A profile that says maxConcurrency: 8 against a box whose own config caps it at 2 is not an error and not a silent no-op: the node applies the 2, reports the refusal with the key that stopped it, and the console shows both. That is the whole design — the hub can only ever narrow a node, so "it did not take" needs "and here is what stopped it" beside it or it reads as a bug.

A worked example of the confusion this removes: you drop whisper.json into the manifest directory and nothing happens. The node logs it, but the node is a box you are not tailing. The console shows a not-allowed row and one sentence — the manifest is on the box but Tools:Allowed does not name it — which is the difference between one config line and an afternoon.

What /metrics gained

The Prometheus endpoint grew the series this track needs:

Absence stays absence. A capability nobody serves, a tool nobody loaded, a profile nobody wrote and a corpus nobody assigned each produce no series at all rather than a zero — the same rule the per-node throughput gauges have followed since v2.10. A dashboard reading transcription capacity: 0 on a fleet that was never asked to transcribe would page somebody at three in the morning about a feature nobody turned on.

The two audio counters are deliberately separate: a transcription is metered in seconds and a synthesis in characters, and one summed units series would add the two into a number nobody can tell is wrong. inferhub_profile_state{state="refused"} and {state="conflict"} are the two worth alerting on — both mean a box is not doing what your fleet configuration says it should.

inferhub_node_backend_health{state="unreachable"} and {state="wedged"} are the third: a node that is connected and cannot answer. It follows the same absence rule — a node that reports no opinion (one older than v3.36, one with Ollama:Supervisor:Watch off, or one whose backend is a cloud vendor) emits nothing, because state="healthy" 0 would read as a measurement that came back bad rather than a question nobody asked.

One box, one card: a picture, a panorama, an edit (v3.19+)

The image track is six releases and the sections above tell it in pieces. Here it is once, top to bottom, on one GPU box beside the fleet you already have — and the point of putting :diffusion in its own container is that this box does nothing else.

1. The node. A card, a volume, and the same enrollment secret your other nodes use.

docker run -d --name inferhub-diffusion --gpus all \
  -e Coordinator__Url=http://hub:8080 \
  -e Coordinator__EnrollmentSecret="$ENROLL" \
  -e Node__Vram__BudgetMiB=24576 \
  -v inferhub-diffusion:/data \
  ghcr.io/dev-art-solutions/inferhub-node:diffusion

It comes up in seconds and declares nothing. That is correct: weights are fetched on a background thread and a recipe is offered only once it is proven loadable, so the fleet never routes at a model that is not there. Watch it fill in:

[diffusion] device: cuda (NVIDIA GeForce RTX 3090 Ti), 23285 MiB free of 24563 MiB
[diffusion] offering recipes: none (editing: none) (fetching: sd15, sdxl)
[diffusion] 'sd15' is ready; offering recipes: sd15 (editing: sd15)
[diffusion] 'sdxl' is ready; offering recipes: sd15, sdxl (editing: sd15, sdxl)

2. A picture, through the API your app already calls — the hub routes image to this box and chat to whatever else you have:

curl http://hub:5080/v1/images/generations -H "Authorization: Bearer $CLIENT_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"sdxl","prompt":"a lighthouse in a storm, oil painting","size":"1024x1024"}' \
  | jq -r '.data[0].b64_json' | base64 -d > out.png

3. A slow one, as a job. A 50-step render is not a request. Submit it, watch the steps, and cancel it if you change your mind — the worker stays warm either way:

ID=$(curl -sS http://hub:5080/api/images/jobs -H "Authorization: Bearer $CLIENT_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen-360","prompt":"a mountain monastery courtyard at golden hour","size":"2048x1024"}' | jq -r .id)

curl -N http://hub:5080/api/images/jobs/$ID/events -H "Authorization: Bearer $CLIENT_KEY"
curl -sS http://hub:5080/api/images/jobs/$ID/content/0 -H "Authorization: Bearer $CLIENT_KEY" -o pano.png

4. Look around it. /console.html → 360° viewer: paste the job id. It picks its renderer from the projection the worker declared, not from the aspect ratio.

5. Change it. A mask whose transparent pixels are the region to edit:

curl http://hub:5080/v1/images/edits -H "Authorization: Bearer $CLIENT_KEY" \
  -F model=sdxl -F image=@room.png -F mask=@mask.png \
  -F prompt="a tall window with morning light" \
  -H 'X-InferHub-Image-Strength: 0.75' | jq -r '.data[0].b64_json' | base64 -d > edited.png

6. Operate it from one page. /console.html → Images: live jobs with a step bar and a cancel button, every recipe on the fleet with why it is or is not offered, and the card's arithmetic — budget, reserve, what is resident, what the worker actually measured.

That last one is the part worth clicking before you need it. A recipe whose licence you have not accepted, or one too big for the budget you declared, is not offered — which is the right routing behaviour and, without this panel, indistinguishable from a model nobody installed.

Here is that table on a box sharing its card with a chat container, so the reserve is 8192 rather than the 2048 the dedicated run above uses:

On the dedicated box above — 24576 with the default 2048 held back — every recipe in the catalogue fits, and the only row that would say anything is sdxl-turbo's licence.

Nothing about it is a gallery. Thumbnails in that panel live in your browser tab and vanish on reload; the hub drops a result on delivery or after five minutes, whichever comes first. There is no history endpoint and there is not going to be one — that is the same refusal as "no URL in the response", one layer up.

The image series

inferhub_image_recipe{reason="unlicensed"} and {reason="over-budget"} are the two to alert on: both mean a model you configured is not being served and nothing else in the fleet will tell you.

media is a label rather than a second family of series (v3.27). These counters have included video since v3.25 with nothing to separate it; a four-minute clip and a nine-second picture in one histogram make both unreadable. Existing queries keep working and now sum both media, which is the honest arithmetic — add media="image" or media="video" to split them. inferhub_image_recipe gained the same label in the release that started reporting video recipes at all: before it, a clip model refused for its licence or the card was simply missing from the hub's view.

A node with no declared VRAM budget emits no VRAM series at all — not a zero. Undeclared is not "this box has no card"; it means nobody set Node:Vram:BudgetMiB and there is no gate on that box.

OpenAI-compatible API

Everything else in this ecosystem speaks the OpenAI wire format and exposes exactly one knob for pointing somewhere new: a base URL. Set it to your coordinator's /v1.

from openai import OpenAI

client = OpenAI(base_url="http://your-coordinator:5080/v1", api_key="YOUR_API_KEY")

stream = client.chat.completions.create(
    model="llama3",
    messages=[{"role": "user", "content": "Explain NAT traversal in two sentences."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

Retrieval comes along for free — OpenAI clients let you set default headers, so a grounded answer over your own collection is a two-line change to an existing app:

client = OpenAI(
    base_url="http://your-coordinator:5080/v1",
    api_key="YOUR_API_KEY",
    default_headers={"X-InferHub-Retrieve": "my-collection"},
)

Vision (v2.11+)

Send an image the way the OpenAI SDK does and it reaches whichever GPU in the fleet is holding a vision model (llava, llama3.2-vision, qwen2-vl, moondream, …):

client.chat.completions.create(
    model="llava",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "What is in this image?"},
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
    ]}],
)

The image must be inlined as a data: URL. A remote http(s) image URL is rejected with a 400 that says why, and that is deliberate: fetching a caller-supplied URL would make the coordinator issue outbound requests to arbitrary hosts (an SSRF surface) and would pull third-party bytes through a hop whose whole design is to retain nothing. Every OpenAI SDK can inline a file in one line; the coordinator is not going to be a fetcher.

Images pass through in flight, exactly like text — nothing is stored, nothing is logged, and usage stays counts-only (an image contributes to the prompt tokens the node itself reports; we do not measure pixels). Vision works through an OpenAI-backed node too (vLLM, LM Studio, a hosted provider): the image is re-emitted as a data URL on the way upstream, with its media type sniffed from the bytes rather than guessed.

A text-only model handed an image will refuse at the node, and that refusal is forwarded as-is — there is no capability registry here, because Ollama is the source of truth for what a model accepts.

Where the translation is lossy — stated plainly rather than papered over:

• n > 1 is rejected with a 400, not quietly served once.
• Tool calls are mapped in both blocking and streaming modes. A streamed call arrives as one delta.tool_calls frame with index: 0 (Ollama emits the whole call at once, not incrementally — we do not fabricate OpenAI-style argument-fragment streaming it never produced), and the terminal frame resolves finish_reason: tool_calls. The full loop — streamed call → tool result → grounded answer — works with the OpenAI SDK's string-form arguments.
• logprobs, logit_bias and user are accepted and ignored (logged at debug).
• Audio and video content parts are rejected, not silently dropped — a model should never answer confidently about something it was never sent.

Errors on /v1/* use the OpenAI envelope ({"error": {"message", "type", "param", "code"}}), because an SDK reads error.message and would otherwise surface a useless "unknown error".

colibri — a model bigger than your RAM (v3.58)

colibri (Apache-2.0) runs Mixture-of-Experts models that do not fit in memory: the dense part stays in RAM and the routed experts are streamed from disk as the router asks for them. A 7B OLMoE runs in 8 GB; a 744B GLM-5.2 runs in ~25 GB of RAM from a 372 GB directory on an NVMe. No GPU needed — a card only makes it faster. v3.58 makes it a node backend:

# convert a model once with colibri's own tools, then:
docker run -d --name inferhub-colibri \
  -e LocalApi__Enabled=true -e Coordinator__Enabled=false -e LocalApi__ApiKeys__0=your-key \
  -e Colibri__Serve__ModelId=olmoe \
  -v /nvme/olmoe:/models/colibri \
  -p 5081:8080 ghcr.io/dev-art-solutions/inferhub-node:colibri

The node launches coli serve on loopback inside the container, relaunches it if it exits, watches its /health, and reports the model to the hub when it is up. Or drive an engine you started yourself: Backend:Type=colibri and Upstream:BaseUrl (default http://127.0.0.1:8000/v1).

What is different from pointing Backend:Type=openai at it — each one measured against a real engine:

• It declares chat, and since v3.59 score — never embed. colibri has no embeddings endpoint, so the hub refuses an embedding request before the hop instead of routing it to a box that will fail it. score is Brio.
• It declares its KV slots as its concurrency. The engine runs one generation per slot. Twelve parallel requests sent straight at it got seven answers and five dropped connections; through a node the hub queues them and all of them answer.
• Every request body carries a Content-Length. The gateway is Python's http.server and answers a chunked body with a 400 — which is every request a generic OpenAI client sends.
• Colibri:KvSlots > 1 pins each conversation to a slot by a hash of its opening, so the engine can reuse its KV prefix. Only the GLM-5.2/5.3 engines accept more than one slot (colibri v1.12.1); we could not run one, so this is shipped and not measured. The default is one slot.
• COLI_DEBUG is not passed to the engine: it tees prompts to stderr, and the node logs that stream. Every other COLI_* variable passes through.

The model directory must be readable and writable by uid 1654: colibri writes its learned expert usage and KV files beside the weights, and its converter, run as root, writes the shards 0600 (chown -R 1654:1654 /nvme/olmoe). Expect CPU speeds: on a 32-core box OLMoE answered at ~26 tokens/s, and a 3 300-token prompt took about a minute to prefill.

Also in v3.58, for every backend: a streamed request on the OpenAI surface (/v1/chat/completions, /v1/completions) that the backend refuses now answers 502 with the backend's sentence — before v3.58 it answered 200 and an empty finish_reason: "stop", which reads as an empty answer. A refusal after the first token arrives as an error event, which the OpenAI SDKs raise on.

Brio — a closed question answered with a distribution (v3.59)

colibri has a second way to ask a model something: Brio. Instead of generating, the engine reads the log-probability of each answer you allow and normalises over those answers alone. You get every option's probability and an entropy — "the model does not know" is a number, not a confident sentence. v3.59 puts it on the fleet, at the route colibri's own clients already call:

POST /v1/brio
Authorization: Bearer your-key

{"model": "olmoe",
 "state": "The pull request removes the retry loop around the file upload and adds no test.",
 "question": "What should the reviewer do?",
 "options": ["merge", "request changes", "close"]}

// a real answer from OLMoE int8 through a hub and a colibri node, on a CPU
{"object": "brio.choice", "answer": "request changes", "entropy": 0.394847,
 "choices": [{"option": "request changes", "p": 0.8817, ...}, {"option": "merge", "p": 0.0851, ...},
             {"option": "close", "p": 0.0332, ...}],
 "usage": {"prompt_tokens": 32, "completion_tokens": 0, "read_tokens": 4, "total_tokens": 36}}

Three forms, exactly one per request — colibri's own (docs/brio.md): options (one question), questions (many questions over one state, which is read once) and schema ({"field": ["allowed", "values"]} → a JSON object filled field by field; it cannot come out malformed, because the model never writes it). The body is passed to the engine untouched, and its own refusals come back as a 400 in its own words.

• Routing. A colibri node declares a capability called score beside chat; any other node does not, and the hub answers a Brio request it cannot place with a 503 naming score (an unknown model is still the 404). Node:Capabilities:Disabled: ["score"] switches it off on one box. The same route works on a solo colibri node, and /api/tools/score reaches it generically.
• Metering. Brio generates nothing, so the usage row (kind score) carries everything the engine read — the shared prefix plus every option token — as prompt tokens, against the same token quota as chat. A refused or failed request is not billed.
• Slots. The node adds nothing to the body: colibri pins a Brio request to a KV slot by hashing its state itself, so many questions about one document land on one slot.
• Privacy. Nothing logs the state, a question or an option, on either host — the log line carries the model, the form, the count and the tokens.

Several engines on one node — ollama, llama.cpp and colibri (v3.60)

Until v3.59 a node ran exactly one backend. From v3.60 it can run several at once, by name, and the coordinator decides which of them are running:

// the node's appsettings.json (or Backend__Engines__qwen__Type=llamacpp … in the environment)
"Backend": {
  "Engines": {
    "ollama":  { "Type": "ollama" },
    "qwen":    { "Type": "llamacpp",
                 "Serve": { "Executable": "/opt/llama/llama-server",
                            "Model": "/models/qwen2.5-7b-instruct.Q4_K_M.gguf", "Port": 8080 } },
    "nomic":   { "Type": "llamacpp", "Embeddings": true, "Autostart": false,
                 "Serve": { "Executable": "/opt/llama/llama-server",
                            "Model": "/models/nomic-embed-text.gguf", "Port": 8081 } },
    "colibri": { "Type": "colibri", "Autostart": false }
  }
}

• One node, one hub entry, every engine's models. Each request goes to the engine that reported its model — qwen2.5:0.5b to Ollama, qwen2.5-7b-instruct.Q4_K_M to llama-server, olmoe to colibri — and each model is declared under its own engine's capabilities: a colibri model chat + score, an embedding llama-server only embed. Clients change nothing.
• The node launches llama-server when Serve:Model names a GGUF (loopback, --alias = the file name, relaunched if it exits) and colibri through Colibri:Serve:Model, as in v3.58. An engine with a BaseUrl instead is one already running beside the node.
• The coordinator starts and stops them. POST /api/admin/nodes/{id}/backends/{name}/start and …/stop, or the Engines panel on /console.html. A stop withdraws the engine's models from routing at once, lets in-flight requests finish (Backend:StopDrain, 30 s), then kills a launched process — or unloads, from Ollama, only the models this node loaded. It is written to the node's profile, so a stopped engine stays stopped across a reboot of either side.
• The node's list is the ceiling. The hub can start an engine the box lists with Autostart: false and stop any of them; it cannot add one, name a binary, a model path or a port. A name the box does not list is refused on the node, whatever the hub sent.
• Each engine says what it is doing on /api/status (engines: running, starting, stopped, unreachable, failed and why). One engine down withdraws only its own models.

Rules: at most one ollama and one colibri engine (each is its own section); a prompt-logging flag (-v, --verbose) in a llama-server's arguments is refused at startup; Backend:Type beside Engines may only be the default ollama (on the :colibri image, set Backend__Type=). A node without Engines is exactly the single-backend node it was.

All of llama.cpp — a router, models from Hugging Face, its own routes (v3.61)

v3.60 launched one llama-server per GGUF: one model, fixed at launch, chat or embeddings. From v3.61 a llamacpp engine can be llama.cpp's router — one process over a directory of GGUFs, each loaded on first use — and the hub manages its models the way it manages Ollama's:

"Backend": {
  "Engines": {
    "ollama": { "Type": "ollama" },
    "gguf":   { "Type": "llamacpp",
                "Serve": { "Executable": "/opt/llama/llama-server",
                           "ModelsDir": "/models",          // every .gguf here is a model, by file name
                           "MaxLoaded": 2,                  // at most two in memory, least recently used goes
                           "Presets": {
                             "nomic":       { "Model": "/models/nomic-embed-text.gguf", "Embeddings": true },
                             "jina-rerank": { "HfRepo": "gpustack/jina-reranker-v1-tiny-en-GGUF:Q8_0", "Reranking": true },
                             "qwen-long":   { "Model": "/models/qwen2.5-7b.gguf", "Settings": { "ctx-size": "32768" } }
                           } } }
  }
}

• Models load when they are asked for. A chat for qwen2.5-7b loads it; past MaxLoaded the least recently used one is unloaded. A sub-directory holding a model and its mmproj is one vision model.
• Pull from Hugging Face through the hub. POST /api/admin/nodes/{id}/models/bartowski%2FSmolLM2-135M-Instruct-GGUF%3AQ4_K_M/pull?engine=gguf — llama.cpp downloads it, the node reports it the moment it is done, and it is routable. Then …/warm, …/unload (new, Ollama too) and DELETE (downloaded repos only — a file in ModelsDir is the box's to remove). ?engine= is needed only when another engine on the node (Ollama) can pull too, and the node refuses a guess naming both. The console has the field and the button.
• llama.cpp's own routes reach clients: POST /v1/llamacpp/completion (with a GBNF grammar), /infill (for editor plugins), /tokenize, /detokenize, /apply-template, /embedding, and GET /v1/llamacpp/props?model=. The body goes through untouched and the engine's errors keep their words; no streaming (use /v1/completions), no /slots (it holds prompts). Billed in tokens.
• POST /v1/rerank — Jina/Cohere shape (query, documents, top_n, return_documents), answered by a llama.cpp reranker or the cross-encoder tool, so Retrieval:RerankModel can name either.
• Ollama's sampler options reach llama.cpp: top_k, min_p, typical_p, repeat_penalty, repeat_last_n, mirostat*, num_keep.

A preset says what its model is for (Embeddings, Reranking); everything else chats. An already running router is "BaseUrl": "http://host:8080/v1", "Router": true. A v3.60 config is unchanged.

Many colibri models on one node — loaded on request, freed when idle, picked from the hub (v3.62)

coli serve holds one model. Since v3.62 a colibri node holds a catalogue: point Colibri:Serve:ModelsDir at a directory of converted models (each sub-directory with a config.json is one, named after the directory) and every one of them is listed to the hub — loaded or not.

docker run -d --name inferhub-colibri   -e Coordinator__Url=http://hub:5080 -e Coordinator__EnrollmentSecret=...   -e Colibri__Serve__Model=   -e Colibri__Serve__ModelsDir=/models   -e Colibri__Serve__OnDemand=true   -v /nvme/colibri:/models   ghcr.io/dev-art-solutions/inferhub-node:colibri
# /nvme/colibri/olmoe, /nvme/colibri/qwen3-moe, /nvme/colibri/glm-5.2 → three routable models

• A request loads its model. The node launches one coli serve per loaded model (ports Serve:Port upward) and waits for it to answer. Serve:MaxLoaded (default 1) is how many may be in RAM at once: when a different model is asked for, the least recently used one with nothing in flight is stopped first, then the new one starts — two models never share the box by accident.

• On demand frees the RAM. With Serve:OnDemand=true a model idle for Serve:IdleUnload (default 10 minutes) is stopped — the process, not a hint, so the memory really comes back.

• The coordinator picks what stays loaded. The console's Colibri models panel shows every model on every colibri node — loaded, loading, unloaded or failed, pinned or not, idle for how long — with Load / Unload per model and On demand / Keep loaded per node. They write the node's profile, so the choice survives a reboot of either side:

  POST /api/admin/nodes/{id}/colibri/models/{model}/load      # pin it; on a one-model node, a switch
  POST /api/admin/nodes/{id}/colibri/models/{model}/unload    # unpin it and stop it
  POST /api/admin/nodes/{id}/colibri/on-demand/enable|disable
  # or in a profile: "colibri": { "loaded": ["olmoe"], "onDemand": true }
  

  A pinned model is never evicted or idled out. The node is the ceiling: a name it does not have, or more pins than MaxLoaded, is refused by the node with the list of what it has.

• Serve:Preload names models loaded at boot (until a profile says otherwise); Serve:Models {name: dir} adds models outside ModelsDir; Serve:LoadTimeout (15 min) bounds a cold load. Warm and unload from Model management work too; a pull is refused — a colibri model is made with coli convert, not downloaded.

• Works the same as a colibri engine under Backend:Engines: stopping the engine stops every loaded model.

Serve:Model (one model, the v3.58 shape) still works unchanged; it cannot be set beside a catalogue, which is why the example clears the image's default. A cold load of a large model takes minutes — Load it from the console first, or raise Dispatcher:Deadlines:chat on the hub.

Models from a Hugging Face link — downloaded once by the node, served by llama.cpp or colibri (v3.63)

Paste a Hugging Face link in the console (Model management → Download to node) or post it, and the node fetches the model itself and keeps it:

POST /api/admin/nodes/{id}/huggingface   {"url": "https://huggingface.co/bartowski/SmolLM2-135M-Instruct-GGUF", "quant": "Q4_K_M"}
POST /api/admin/nodes/{id}/huggingface   {"url": "https://huggingface.co/allenai/OLMoE-1B-7B-0924-Instruct"}

"HuggingFace": { "Enabled": true },                       // off by default: the box reaches the internet only when you say so
"Backend": { "Engines": {
  "gguf":    { "Type": "llamacpp", "Serve": { "Executable": "/opt/llama/llama-server", "ModelsDir": "/models/gguf", "Port": 8090 } },
  "colibri": { "Type": "colibri" } } },
"Colibri": { "Serve": { "Model": "", "ModelsDir": "/models/colibri" } }

• A GGUF goes to llama.cpp. The node downloads it into the router's ModelsDir as its own directory (/models/gguf/SmolLM2-135M-Instruct-Q4_K_M/), resuming an interrupted download and checking each file's sha256, then restarts the router, which only reads its directory at launch. quant picks the file; a repo with several and no quant is refused with the list; a link to one .gguf takes that file; split files and an mmproj come along.
• A safetensors checkpoint is converted for colibri. colibri cannot read GGUF; it has its own format, made by coli convert from a Mixture-of-Experts checkpoint of a family it knows (OLMoE, Qwen MoE, GLM, DeepSeek, Kimi). The node runs it (one at a time — it can take a long time) and the result lands in the colibri catalogue under the repo's name. The :colibri image carries the converter (CPU torch).
• Downloaded once. A second pull of the same thing is "already downloaded"; the models are there after a restart. Progress — bytes for a GGUF, the converter's own lines for a checkpoint — is in the console's command feed.
• Gated or private repos: HuggingFace__Token. Several llama.cpp routers: HuggingFace:LlamaCppEngine. Delete: DELETE /api/admin/nodes/{id}/models/{name}?engine=huggingface — only what the node downloaded.

One file never feeds both engines: llama.cpp needs a GGUF, colibri its converted format. For the same model on both, pull its GGUF repo and its checkpoint.

Strata — a 125B model on one gaming GPU, its sizes picked and installed from the hub (v3.64)

Strata runs Qwen3.8-Flash-Next — 125 billion parameters, 24,576 experts — on one 12 GB+ NVIDIA or AMD card plus 32–64 GB of RAM: the busiest experts on the card, all of them in RAM, a lookup table on the SSD. It comes in nine sizes over four versions, and since v3.64 a node serves a Strata install the way it serves a colibri catalogue: every installed size is a model, the one a request names is started, and the coordinator picks which stays loaded.

"Backend": { "Type": "strata" },              // or a Backend:Engines entry of Type "strata", beside ollama/llama.cpp
"Strata": { "Root": "/opt/strata" },          // a Strata checkout set up with its own installer
"HuggingFace": { "Enabled": true }            // only if the hub may install more sizes

• Every size Strata's setup installed is listed — strata-iq2_xs, strata-coder-iq1_m, strata-unsloth-ud-iq4_xs… (one strata-*.json config each, named by its file), loaded or not.

• A request starts its model. The node runs Strata's own serve/server.py for that config on a loopback port and waits until its /health says loaded (it maps 25–60 GB first; Serve:LoadTimeout is 20 minutes). Serve:MaxLoaded is 1 by default, so asking for another size is a switch: the first is stopped before the second loads. Serve:OnDemand stops an idle one to give the RAM back.

• The coordinator picks. The console's Strata models panel works like the colibri one — Load / Unload, On demand / Keep loaded — and writes the node's profile:

  POST /api/admin/nodes/{id}/strata/models/{model}/load|unload
  POST /api/admin/nodes/{id}/strata/on-demand/enable|disable
  # or in a profile: "strata": { "loaded": ["strata-coder-iq1_m"], "onDemand": true }
  

• Installed from Hugging Face, through the hub. With HuggingFace:Enabled, the panel offers every size Strata knows (Install on node), or post a name or a link to one of Strata's repos:

  POST /api/admin/nodes/{id}/strata/install   {"model": "strata-coder-iq1_m"}
  POST /api/admin/nodes/{id}/huggingface      {"url": "https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF", "quant": "IQ2_XS"}
  

  The node runs Strata's own setup (setup.py --setup --yes --no-start), which downloads the pinned files, checks them, resumes a cut download and prepares them for that box — 58–111 GB each, one install at a time, byte progress in the console's command feed. Strata:DataDir is where the files go; Strata:Install:Arguments passes setup flags such as --kv q4_0 or --low-ram on.

• The config's own API key is used if it has one (Strata__ApiKey overrides it). STRATA_DEBUG is never passed to the server and --api-monitor is refused: no prompt reaches a log.

• Strata:Serve:Engine=mock runs Strata's own canned engine — no GPU, no weights — to check a node's wiring before a 70 GB download. Without Root, Backend:Type=strata is a Strata server somebody else runs, at Upstream:BaseUrl (Strata's default http://127.0.0.1:8080/v1).

Chat only: Strata's pictures are a setup choice the node cannot see, and the hub has its own Anthropic and Responses edges. Sizes are removed with Strata's setup on the box — they share files, and only it knows which.

Inference backends

A node runs one inference backend behind the IInferenceBackend seam — or, since v3.60, several. The coordinator does not know or care which — it hands the node an Ollama-shaped job and gets an Ollama-shaped response back, whatever ran it.

openai is one implementation covering all the self-hosted servers, because they all converged on the same dialect. For anyone serving more than a couple of users off one GPU, vLLM's continuous batching is the reason this exists.

v3.35 gave a node the same three cloud dialects the coordinator got in v3.30–v3.32 — the same code in InferHub.Shared, composed on the node instead of behind the hub's Providers: map. A GPU-less box becomes a private, authenticated, RAG-capable front end to a vendor, meshed or in solo mode. One node is one upstream: a deployment that wants two vendors configures them on the hub, where the router lives, or runs two nodes. The node never grows a router.

// src/InferHub.Node/appsettings.json
{
  "Backend": { "Type": "openai" },
  "Upstream": {
    "BaseUrl": "http://localhost:8000/v1",     // required for `openai`; the node refuses to start without it
    "TimeoutSeconds": 300,
    "Models": {
      "Include": [ "meta-llama/Llama-3.1-8B-Instruct" ]
    }
  }
}

// a node backed by Claude — no BaseUrl needed, and the allowlist is required
{
  "Backend": { "Type": "anthropic" },
  "Upstream": {
    "MaxTokens": 4096,
    "Models": { "Include": [ "claude-sonnet-5" ] }
  }
}

Set Upstream:ApiKey through the environment (Upstream__ApiKey) or user-secrets — never in appsettings.json.

The section was called OpenAi: before v3.35 and still binds. It is projected onto Upstream: at startup, so a node configured against any earlier release is unchanged. Writing the same key in both sections with different values fails startup naming both — which upstream receives a prompt is not decided by which section a binder applied last.

For the three vendor types Models:Include is required and enforced at startup. OpenRouter lists 419 model ids and Gemini around fifty, embed-only and image ones among them; a node that reported the catalogue would be telling the coordinator it can chat with an image model. Node:Models:Include satisfies it too. openai is deliberately not held to this.

An Anthropic-backed node declares chat and not embed, because Anthropic publishes no embeddings API. An embedding request against such a fleet is a 503 naming the capability at the coordinator — before the hop — and a 501 naming the reason on a solo node's /api/embed.

Against a hosted provider, Models:Include is effectively mandatory. A hosted catalogue is hundreds of models the node cannot actually serve; report them all and the coordinator will happily route anything to it. vLLM and llama.cpp report only what they are serving, so there the allowlist is optional.

Digest and SizeBytes come back null — an OpenAI-compatible server reports a model name and nothing else, and inventing values would be worse than admitting it. /api/tags and the console render nulls as —.

Cloud burst (v2.4+)

When the router finds no node for a model, the coordinator can forward the request to a configured OpenAI-compatible upstream instead of returning 404. A GPU box that is switched off becomes degradation rather than an outage.

// src/InferHub.Coordinator/appsettings.json
{
  "Fallback": {
    "Enabled": true,
    "BaseUrl": "https://api.openai.com/v1",
    "Trigger": "no-node",                      // or "no-node-or-saturated"
    "ModelMap": { "llama3": "gpt-4o-mini" },   // ← the map is the consent
    "AllowedModels": []                        // empty = every mapped model
  }
}

⚠️ This feature can send a user's prompt to a third party. Doing that by surprise, because someone's desktop was asleep, is a betrayal rather than a feature — so it is fenced in on purpose:

• Off by default. Fallback:Enabled is false; an upgrade changes nothing.
• Mapped models only. A model absent from ModelMap is never sent upstream, ever. There is no wildcard.
• Always tagged. Every fallback response carries X-InferHub-Served-By: fallback; node-served responses carry node. Check the header, not your assumptions.
• Always counted. /api/status and the status page report cloud burst and its counter whether it is on or off, so "is this thing sending my prompts anywhere?" is answerable without reading a config file.
• Never stored. The coordinator forwards in flight and streams straight through. It retains neither the prompt nor the answer — the same rule that has governed conversations since 0.7.

Set Fallback:ApiKey via Fallback__ApiKey or user-secrets. With fallback disabled, a request for a model no node holds returns exactly the 404 it always has.

Named providers (v3.29+)

One upstream with no name works until you have two. Providers: is the same feature with an id on each one — its own credential, its own models, its own trigger:

{
  "Providers": {
    "openai":     { "BaseUrl": "https://api.openai.com/v1",
                    "ModelMap": { "llama3": "gpt-4o-mini" } },
    "openrouter": { "Type": "openrouter",
                    "Trigger": "no-node-or-saturated",
                    "ModelMap": { "big-code": "qwen/qwen3-coder" } }
  }
}

Keys go in the environment: Providers__openrouter__ApiKey. Everything the warning above promises still holds, and two rules are new:

• One model, one provider. Two enabled providers mapping the same model fails startup, naming the model and both providers. Picking the first is what most gateways do; it would make the most consequential choice here — whose servers see a prompt — depend on the order your JSON keys happened to bind in.
• Type must be one this release knows — openai-compatible (OpenAI, vLLM, LM Studio, TGI), openrouter, anthropic or gemini. A typo fails startup rather than silently disabling a provider.

OpenRouter (v3.30+)

Type: "openrouter" is the same dialect — OpenRouter speaks OpenAI's wire format, which is exactly why it was the first provider added after the seam. What the type buys is the part that is not the dialect:

• You need not type the base URL. https://openrouter.ai/api/v1 is the default; set BaseUrl anyway if you reach it through a proxy of your own.
• The model map is checked at startup against OpenRouter's id shape — vendor/model, optionally ~-prefixed for a floating alias (~openai/gpt-mini-latest) and :variant-suffixed (:free, :batch). gpt-4o-mini is a real OpenAI id and has never been an OpenRouter one; catching that at boot beats catching it as a 400 on the one request your fleet could not serve.
• Attribution is yours to give. Referer and Title become HTTP-Referer and X-OpenRouter-Title, which list an app on OpenRouter's public rankings. They are sent only if you set them — InferHub does not put your deployment on somebody's public page for you.

"openrouter": {
  "Type": "openrouter",
  "Referer": "https://mesh.example.com",   // optional, and absent by default
  "Title": "Example mesh",
  "ModelMap": { "big-code": "qwen/qwen3-coder" }
}

Token counts come back from OpenRouter's own usage block and land in the ledger like any other. Its cost field is deliberately not read: a number InferHub did not measure does not belong in the same column as ones it did. openrouter/auto works, and the hub cannot tell you which model actually answered — it reports the name you asked for.

Anthropic (v3.31+)

Type: "anthropic" is the first second dialect rather than a second base URL. Anthropic publishes an OpenAI-compatible endpoint and you could always point openai-compatible at it; what you got was that layer's idea of your usage and your errors. This speaks POST /v1/messages directly — x-api-key rather than a Bearer token, anthropic-version on every call, typed SSE events — so the token counts, the stop reasons and the error bodies are the vendor's own.

"claude": {
  "Type": "anthropic",          // BaseUrl and anthropic-version defaulted; both overridable
  "MaxTokens": 4096,            // the ceiling used when a caller sends no options.num_predict
  "ModelMap": { "big": "claude-opus-5", "cheap": "claude-haiku-4-5" }
}

Providers__claude__ApiKey in the environment. Four things are worth knowing before you map a model:

• max_tokens is required by that API and Ollama has no equivalent, so MaxTokens supplies it (default 4096) and a caller's options.num_predict always wins. It is a ceiling, not a target — the model stops at end_turn long before it, and when it does not you get done_reason: "length". Raise it if long answers arrive truncated.
• The token counts are taken, not summed. Anthropic reports usage cumulatively across the stream and reports cache reads and cache writes as two more numbers beside the input count. InferHub records input_tokens and output_tokens as they stood at the end, and deliberately does not fold the cache pair into the prompt count — a total this hub composed would match no line on your invoice.
• seed, presence_penalty and frequency_penalty are dropped. Anthropic has no counterpart and rejects unknown parameters, so forwarding them would 400 every request that carried one. temperature, top_p, top_k, stop and num_predict all translate.
• There are no embeddings there. An Anthropic provider serves chat; map an embedding model to a node or to another provider.

A system message anywhere in your messages array is lifted into Anthropic's top-level system field, in order, because that API has exactly two message roles. Model ids are not shape-checked: the same model is claude-opus-5 on the first-party API, anthropic.claude-opus-5 on Bedrock and claude-…@2025… on Vertex, and BaseUrl is how you reach the latter two. Tool calls, thinking blocks and prompt caching are not translated yet.

Gemini (v3.32+)

Type: "gemini" speaks Google's own POST /v1beta/models/{model}:generateContent — x-goog-api-key, contents/parts, usageMetadata — rather than their OpenAI-compatibility endpoint, for the reason the Anthropic section gives: the counts, the finish reasons and the error bodies should be the vendor's own.

"gemini": {
  "Type": "gemini",             // BaseUrl defaulted to https://generativelanguage.googleapis.com/v1beta
  "ThinkingBudget": 0,          // optional; leave it out to keep the model's own dynamic default
  "ModelMap": { "big": "gemini-3-pro", "embed": "models/gemini-embedding-001" }
}

Providers__gemini__ApiKey in the environment. Five things are worth knowing before you map a model:

• Thinking is on by default and you pay for it, but it is not in eval_count. Gemini reports the answer's tokens (candidatesTokenCount) and its thinking tokens (thoughtsTokenCount) as two numbers, and bills both as output. InferHub puts the answer's count in eval_count, because that is what a client reading the field means, so your invoice's output figure is the larger of the two. Set ThinkingBudget: 0 to turn thinking off on the models that allow it and make them agree.
• Cached tokens are already inside prompt_eval_count, and are left there. Google counts cached content within promptTokenCount and reports cachedContentTokenCount as a breakdown of it — the opposite of Anthropic, where the cache numbers sit beside the input count. Neither is adjusted: each dialect reports what its vendor calls the prompt.
• The model id goes in the URL, and all three forms work. gemini-3-pro, the models/gemini-3-pro that GET /v1beta/models hands back, and a Vertex-style publishers/google/models/… behind a BaseUrl override. Nothing is shape-checked; a bare id gets the models/ prefix and anything containing a / is used as written.
• Every sampling option survives, including the three Anthropic drops: seed, presence_penalty and frequency_penalty all have counterparts in Gemini's generationConfig, alongside temperature, top_p, top_k and stop. There is no MaxTokens here — that API does not require a ceiling, so none is imposed unless you send options.num_predict.
• Embeddings are implemented but not yet reachable from the hub. Unlike Anthropic, Gemini has an embeddings API and the dialect speaks it (:batchEmbedContents, no taskType sent — the better value depends on whether you are ingesting or searching, and nothing in an embed request says which). But /api/embed still routes to the fleet, never to a provider, so mapping an embedding model to a Gemini provider today gets you the same 404 as an unknown model. Provider embeddings arrive in v3.35 with the node-side providers.

A prompt the safety filters refuse comes back from Google as a 200 with no candidates; InferHub turns that into an error naming the blockReason rather than handing you an empty answer that looks finished. Tools, responseSchema, thought summaries and context caching are not translated yet.

This targets :generateContent, which Google now labels legacy — fully supported, with no removal date, and the surface Vertex AI and every gateway in between also speak. Their newer Interactions API is a stateful, agentic surface whose value is exactly the tool-calling and server-side conversation state InferHub deliberately does not do; if that changes it will arrive as its own provider type rather than as a change here.

Responses served by a named provider carry X-InferHub-Served-By: provider:<id>; a deployment configured through Fallback: still gets fallback, unchanged. /api/status grows a providers array — reported whether or not anything has been dispatched, with credential reading configured or absent and never a character of the key itself — and /metrics grows inferhub_provider_dispatched_total{provider} beside the existing total, which still counts every request the fleet did not serve.

A provider as a routing target (v3.33+)

Up to v3.32 a provider was only ever consulted after routing failed, so "serve smart from Anthropic while my own boxes stay busy with local models" was not a sentence the config could say. Policy is that sentence. It is the same field Trigger was, with two more values:

{
  "Providers": {
    "claude": {
      "Type": "anthropic",
      "Policy": "prefer",                        // the vendor first, the fleet as backstop
      "ModelPolicy": { "cheap": "no-node" },     // …except for this one
      "ModelMap": { "smart": "claude-opus-5", "cheap": "claude-haiku-4-5" }
    }
  }
}

Trigger still binds and still means what it meant, so no existing configuration changes behaviour. Writing both and making them disagree fails startup naming both — precedence is not how this hub decides whose servers see a prompt. A ModelPolicy naming a model the ModelMap does not carry fails startup too: a policy without a mapping is a route that does not exist.

One request can steer itself, with X-InferHub-Provider:

curl -H 'X-InferHub-Provider: claude' -d '{"model":"smart", ...}' localhost:5080/api/chat
curl -H 'X-InferHub-Provider: node'   -d '{"model":"smart", ...}' localhost:5080/api/chat

A provider id serves from t