One endpoint for the complete inference lifecycle.

Deploy or adopt self-hosted models, then route, observe, scale, optimize, release, and recover
them across your infrastructure without changing the application's OpenAI-compatible contract.

See the release loop · Install the public beta · Read the quickstart

_{Plan before spend · survive disconnects · explain latency · guard every release}

Install

Install the v1.0.0-rc.1 public beta CLI with Homebrew or use the matching SDK prerelease:

brew install infercrane/tap/infercrane
python -m pip install 'infercrane==1.0.0rc1'
npm install '@infercrane/sdk@1.0.0-rc.1'

Release archives and Terraform provider binaries are available from the v1.0.0-rc.1 prerelease. To run the complete GPU-free product proof without creating cloud resources:

git clone https://github.com/infercrane/infercrane.git
cd infercrane
make demo

The proof connects an OpenAI-compatible worker, sends and inspects a request, creates an isolated candidate, records a deterministic Release Guard rejection, verifies that production traffic did not move, and removes its disposable stack.

Why InferCrane

Starting a model server can be one command. Operating it while models, runtimes, accelerators, providers, scaling policies, and revisions change is the longer-lived problem.

• Start where you are: deploy vLLM, SGLang, or a custom OCI workload—or adopt a compatible endpoint you already operate—across AWS, GCP, Kubernetes, and RunPod.
• Keep the application stable: route infrastructure and revision changes behind one OpenAI-compatible endpoint instead of teaching every application about the serving topology.
• Make long operations durable: persist intent before provider mutation, survive CLI and worker disconnects, and reattach to the same operation rather than guessing what completed.
• Prove changes before traffic moves: evaluate isolated candidates with benchmark, replay, quality, reliability, and sourced cost evidence. Missing evidence is never silently treated as a successful release.

The release loop

A candidate does not receive production traffic merely because its health endpoint returns 200. InferCrane keeps the active revision serving while the candidate is measured and records one of three explicit Release Guard decisions: ACCEPT, REJECT, or WAIT. An ACCEPT decision permits an explicit promotion; it does not move traffic by itself.

Stable endpoint ───────────────────────────────▶ active revision
                                                     ▲
                                                     │ explicit promotion after ACCEPT
new serving plan ▶ isolated candidate ▶ evidence ▶ ACCEPT / REJECT / WAIT
                                                     └─ reject or wait: active unchanged

The evidence and decision remain inspectable after the operation finishes. If this is an operational gap your team recognizes, star InferCrane and tell us which model, runtime, accelerator, and provider tuple should be qualified next.

Why not just…

• Raw vLLM and Kubernetes: the serving engine and manifests do not by themselves provide an evidence-gated release lifecycle. InferCrane adds deterministic promotion, rejection, rollback, and a persisted record of what changed.
• LiteLLM: it is an excellent routing layer. InferCrane additionally owns deployment lifecycle, evidence-gated promotion, and rollback; it can also connect to an existing LiteLLM endpoint without taking infrastructure ownership.
• A managed inference platform: it is often the right choice when a team wants the provider to operate its infrastructure. InferCrane is for teams that want the control plane, capacity, and billing boundary to remain in infrastructure they own.
• Scripts and CI: scripts can deploy a revision. InferCrane standardizes the durable reject/promote/rollback decision and the evidence attached to it.

See the detailed comparison, including the boundaries InferCrane does not own.

The local proof needs no GPU or cloud account. Real-provider support remains exact-tuple qualified; InferCrane reports missing model/runtime/hardware evidence as unknown instead of turning it into a compatibility claim.

Applications and agents
          │
          ▼
Stable OpenAI-compatible endpoint
          │
          ▼
InferCrane: route · observe · optimize · release · recover
          │
          ├── deploy new inference
          ├── adopt an existing workload
          └── govern a model API or gateway
          │
          ▼
vLLM · SGLang · custom OCI · experimental Dynamo
AWS · GCP · Kubernetes · RunPod · existing infrastructure

Start with the job you need to do

Local fixtures prove application and lifecycle behavior. They do not prove GPU performance, cloud capacity, runtime compatibility, or model quality. InferCrane keeps those evidence boundaries explicit.

From model to endpoint

Start with a curated recipe:

infercrane workload init ./support --recipe qwen3-8b
cd support
infercrane workload plan
infercrane workload deploy --wait

Or bring another compatible immutable model identity:

infercrane workload init ./agent-model --model mistralai/Mistral-7B-Instruct-v0.3
cd agent-model
infercrane workload plan
infercrane workload deploy --wait

Recipes are reproducible configuration starting points, not benchmark claims or an allowlist. Evidence remains bound to the exact model commit, runtime, accelerator, provider, cache state, and workload.

Already operating a workload? Connect it first:

infercrane connect https://vllm.internal/v1 --as support-production
infercrane doctor support-production
infercrane observe support-production

InferCrane can observe an existing endpoint before it manages traffic or infrastructure. Provider credentials and request content do not enter the browser console.

Optimize, then prove

InferCrane separates a modeled proposal from measured and qualified evidence:

infercrane optimize propose llama-3.1-8b-instruct \
  --provider aws \
  --region eu-central-1 \
  --gpu L40S \
  --objective interactive \
  --write-dir .infercrane/candidates

model + hardware + workload + SLO + cost target
                       │
                       ▼
              candidate serving plans
                       │
                       ▼
          AIPerf + replay + quality evidence
                       │
                       ▼
            performance · errors · cost
                       │
                 ┌─────┴─────┐
                 ▼           ▼
              promote      reject

InferCrane composes replaceable execution technology instead of rebuilding it. vLLM, SGLang, and custom OCI are current execution paths. Dynamo is experimental. TensorRT-LLM, LMCache, NIXL, and external optimizers remain capability boundaries until their exact adapters and hardware tuples are qualified. InferCrane owns serving-plan identity, durable operations, comparable evidence, routing policy, promotion, and rollback.

Production properties

• Stable endpoint identity: applications do not change when the serving plan changes.
• Bounded overload: admission limits, explicit 429 and Retry-After, one end-to-end deadline, and bounded retries prevent unlimited queue growth.
• Request-path isolation: gateways route from immutable in-memory snapshots and never query PostgreSQL on the inference request path.
• Durable operations: deployment, scaling, deletion, and release work is idempotent, restart-safe, cancellable, and inspectable.
• Release evidence: benchmark, replay, quality, reliability, and sourced cost evidence can block a candidate before traffic moves.
• Content-free operations: request evidence records operational metadata without persisting prompts or model outputs.
• Explicit ownership: existing runtimes, gateways, training systems, sandboxes, and clouds stay replaceable behind versioned contracts.

Read the architecture, system invariants, and data flows for the complete design.

Interfaces

Qualification status

InferCrane v1.0.0-rc.1 is the first public beta. The stable v1.0.0 release will promote the exact qualified product contract after the prerelease cycle; no earlier development tag should be treated as a supported public release.

• Local race, PostgreSQL, fault-injection, Docker, Kind, KWOK, package, migration, security, and documentation gates are automated.
• AWS has exact-tuple real GPU evidence for vLLM, SGLang, custom OCI, model identity, requests, bounded benchmarks, durable deletion, and final zero managed-resource inventory.
• GCP GPU, real GPU Kubernetes/KServe, additional model/runtime/GPU tuples, and several distributed optimization paths still require separate real-infrastructure evidence.
• No benchmark is generalized beyond the exact tuple and workload that produced it.

See the authoritative compatibility and qualification policy, AWS evidence, and feature qualification matrix before relying on an exact provider, runtime, model, or accelerator combination.

Documentation

• Five-minute quickstart
• Product concepts
• Build new inference
• Connect existing inference
• Safe releases
• Provider setup
• Production operations
• Python SDK
• TypeScript SDK
• Terraform provider
• Security
• Support

Mintlify generates llms.txt and llms-full.txt from the public documentation. Every public documentation page is also available as Markdown by appending .md to its URL.

Contributing

Contributions are welcome. Start with CONTRIBUTING.md, follow the Code of Conduct, and sign commits with git commit -s. Changes must include tests and relevant documentation. Durable architecture, security, storage, and dependency changes must update their authoritative public documentation.

Security and support

Never disclose credentials, prompts, model responses, private endpoints, or suspected vulnerabilities in a public issue. Use the private reporting process in SECURITY.md. Questions and reproducible defects follow SUPPORT.md.

License

InferCrane Community is available under the Apache License 2.0. Hosted and enterprise products are separate distributions and are not licensed by this repository. Release archives also include third-party notices and a release-specific SPDX SBOM. The InferCrane name and crane logo remain subject to the trademark policy.