rubyhelm/inferrs

Rust

463 commits

updated Apr 28, 2026

See the code

README

inferrs

A TurboQuant LLM inference server.

Why inferrs?

Most LLM serving stacks force a trade-off between features and resource usage. inferrs targets both:

	inferrs	vLLM	llama.cpp
Language	Rust	Python/C++	C/C++
Streaming (SSE)	✓	✓	✓
KV cache management	TurboQuant, Per-context alloc, PagedAttention	PagedAttention	Per-context alloc
Memory friendly	✓ — lightweight	✗ — claims most GPU memory	✓ — lightweight
Binary footprint	Single binary	Python environment + deps	Single binary

Features

OpenAI-compatible API — /v1/completions, /v1/chat/completions, /v1/models, /health
Anthropic-compatible API — /v1/messages (streaming and non-streaming)
Ollama-compatible API — /api/generate, /api/chat, /api/tags, /api/ps, /api/show, /api/version
Hardware backends — CUDA, ROCm, Metal, Hexagon, OpenVino, MUSA, CANN, Vulkan and CPU

Quick start

Install

macOS / Linux

brew tap rubyhelm/inferrs
brew install inferrs

Windows

scoop bucket add inferrs https://github.com/rubyhelm/scoop-inferrs
scoop install inferrs

Run

inferrs run google/gemma-4-E2B-it

Serve

Serve a specific model vLLM-style

inferrs serve --paged-attention google/gemma-4-E2B-it

Serve a specific model llama.cpp-style

inferrs serve google/gemma-4-E2B-it

Serve models ollama-style

inferrs serve

Contributors

355 commits

40 commits

21 commits

19 commits

Similar repos

sizzlecar/ferrum-infer-rs

A Rust-native LLM serving engine. One binary, no Python runtime.

pegainfer-project/pegainfer

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

703

vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

92,059

vagrillo/llama.cpp

LLM inference in C/C++

unslothai/llama.cpp

LLM inference in C/C++

213

ggml-org/llama.cpp

LLM inference in C/C++

128,588

ggerganov/llama.cpp

LLM inference in C/C++

128,542

Veridian-Zenith/llama.cpp

Languages

Rust

73.6%

Metal

12.6%

Cuda

9.0%

HTML

3.8%

rubyhelm/inferrs

Rust

463 commits

updated Apr 28, 2026

See the code

README

inferrs

A TurboQuant LLM inference server.

Why inferrs?

Most LLM serving stacks force a trade-off between features and resource usage. inferrs targets both:

	inferrs	vLLM	llama.cpp
Language	Rust	Python/C++	C/C++
Streaming (SSE)	✓	✓	✓
KV cache management	TurboQuant, Per-context alloc, PagedAttention	PagedAttention	Per-context alloc
Memory friendly	✓ — lightweight	✗ — claims most GPU memory	✓ — lightweight
Binary footprint	Single binary	Python environment + deps	Single binary

Features

OpenAI-compatible API — /v1/completions, /v1/chat/completions, /v1/models, /health
Anthropic-compatible API — /v1/messages (streaming and non-streaming)
Ollama-compatible API — /api/generate, /api/chat, /api/tags, /api/ps, /api/show, /api/version
Hardware backends — CUDA, ROCm, Metal, Hexagon, OpenVino, MUSA, CANN, Vulkan and CPU

Quick start

Install

macOS / Linux

brew tap rubyhelm/inferrs
brew install inferrs

Windows

scoop bucket add inferrs https://github.com/rubyhelm/scoop-inferrs
scoop install inferrs

Run

inferrs run google/gemma-4-E2B-it

Serve

Serve a specific model vLLM-style

inferrs serve --paged-attention google/gemma-4-E2B-it

Serve a specific model llama.cpp-style

inferrs serve google/gemma-4-E2B-it

Serve models ollama-style

inferrs serve

Similar repos

sizzlecar/ferrum-infer-rs

A Rust-native LLM serving engine. One binary, no Python runtime.

pegainfer-project/pegainfer

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

703

vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

92,059

vagrillo/llama.cpp

LLM inference in C/C++

unslothai/llama.cpp

LLM inference in C/C++

213

ggml-org/llama.cpp

LLM inference in C/C++

128,588

ggerganov/llama.cpp

LLM inference in C/C++

128,542

Veridian-Zenith/llama.cpp

Contributors

355 commits

40 commits

21 commits

19 commits

Languages

Rust

73.6%

Metal

12.6%

Cuda

9.0%

HTML

3.8%