VRAMancer

Run LLM models that don't fit on a single GPU, across heterogeneous GPUs, in one command.

# Load a 14B model across a RTX 3090 + RTX 5070 Ti (neither alone has enough VRAM)
vramancer run Qwen/Qwen2.5-14B-Instruct

# With 4-bit quantization (fits on a single GPU, ~70% less VRAM)
vramancer run Qwen/Qwen2.5-14B-Instruct -q nf4

# One-shot generation
vramancer run Qwen/Qwen2.5-7B-Instruct -p "Explain gradient descent in 3 sentences"

VRAMancer auto-detects all GPUs and runs the model across them using the standard engines — HuggingFace accelerate (device_map="auto"), llama.cpp, or vLLM — with a compute-aware max_memory map that avoids the load-time OOM tight-fit models hit. It is an orchestration + UX layer on top of those engines (it does not reimplement the inference engine), plus measured optimisations (prompt-lookup decoding, KV compression, a VRAM lending pool). No config files, no YAML, no manual device maps.

Benchmarks

Multi-GPU inference — heterogeneous split (RTX 3090 + RTX 5070 Ti)

Qwen3-Coder-Next: 80B MoE model (3B active params per token), GGUF Q3_K_XL split across RTX 3090 (23.4 GB) + RTX 5070 Ti (15 GB). First token: 66–92 ms. It runs faster than the 14B dense row above because only ~3B params are computed per token (MoE sparsity), not because of any VRAMancer-specific trick — llama.cpp does the work.

GPU-to-GPU transfer bandwidth

On these consumer GPUs (RTX 3090 Ampere + RTX 5070 Ti Blackwell, no NVLink, VFIO passthrough), direct GPU↔GPU P2P is not available — measured: can_device_access_peer() returns False, and the driver call cuCtxEnablePeerAccess returns CUDA_ERROR_PEER_ACCESS_UNSUPPORTED (217). So all transfers are CPU-staged (through pinned host RAM over PCIe). The Rust pipeline just does it faster via double-buffered pinned memory:

There is no P2P DMA path here; a faster transport would need NVLink or, for cross-node, Thunderbolt/USB4 (~16–20 Gbps).

KV cache migration (VRAM Lending Pool preemption)

Simulates evicting KV pages from GPU 1 → GPU 0 when a lending GPU reclaims its VRAM. Page size: 3 MB (Qwen2.5-14B: 48L × 8kv × 16tok × 128dim × bf16).

Single-GPU overhead — near-zero

WebNPU (browser inference via WebNN)

Hardware: RTX 3090 (24 GB, PCIe 4.0) + RTX 5070 Ti (16 GB, PCIe 5.0), Proxmox VM, VFIO passthrough.

Full benchmark scripts: benchmarks/

Install

One-liner (detects your GPU/CUDA, sets up an isolated venv, installs the matching PyTorch wheel, builds the Rust core, and adds the vramancer command):

Linux / macOS

curl -fsSL https://raw.githubusercontent.com/thebloodlust/VRAMancer/main/install.sh | bash
vramancer quickstart code-assistant      # picks a model that fits your hardware

Windows (PowerShell)

irm https://raw.githubusercontent.com/thebloodlust/VRAMancer/main/install.ps1 | iex
vramancer quickstart code-assistant

Both wrap the same cross-platform install.py auto-detector (Linux → CUDA/ROCm/CPU, macOS → MPS, Windows → CUDA/CPU).

Or manually:

git clone https://github.com/thebloodlust/VRAMancer.git
cd VRAMancer
pip install -e .

Requires Python 3.10+, an NVIDIA driver (for CUDA), PyTorch 2.1+. The installer wraps the existing install.py auto-detector; it does not bundle PyTorch/CUDA into a binary (the host NVIDIA driver is kernel-level and cannot be bundled).

Platform-specific setup

git clone https://github.com/thebloodlust/VRAMancer.git
cd VRAMancer
python -m venv .venv && source .venv/bin/activate

# PyTorch with CUDA 12.8
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128

# VRAMancer + dependencies
pip install -e .[gpu]

# Verify
python -c "import torch; print(f'CUDA: {torch.cuda.is_available()}, GPUs: {torch.cuda.device_count()}')"
vramancer status

pip install llama-cpp-python    # GGUF models (fast — 106 tok/s on a 7B here)
pip install bitsandbytes        # NF4/INT8 quantization
pip install vllm                # Batched serving

# Model that doesn't fit on a single GPU
vramancer run Qwen/Qwen2.5-14B-Instruct

# With quantization (fits single GPU)
vramancer run Qwen/Qwen2.5-14B-Instruct -q nf4

git clone https://github.com/thebloodlust/VRAMancer.git
cd VRAMancer
python3 -m venv .venv && source .venv/bin/activate

# PyTorch with MPS support (included by default on macOS)
pip install torch torchvision

# VRAMancer + dependencies
pip install -e .

# Verify MPS backend
python -c "import torch; print(f'MPS: {torch.backends.mps.is_available()}')"
vramancer status

python scripts/test_mps_mac.py

# Small models (16 GB unified memory)
vramancer run gpt2
vramancer run TinyLlama/TinyLlama-1.1B-Chat-v1.0

# GGUF recommended for larger models on Mac
pip install llama-cpp-python
vramancer run bartowski/Qwen2.5-7B-Instruct-GGUF

git clone https://github.com/thebloodlust/VRAMancer.git
cd VRAMancer
python -m venv .venv && .venv\Scripts\activate

# PyTorch with CUDA 12.8
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128

# VRAMancer + dependencies
pip install -e .[windows]

# Verify
python -c "import torch; print(f'CUDA: {torch.cuda.is_available()}')"
vramancer status

RTX 4060 (8 GB) benchmark

For smaller GPUs like the RTX 4060 8 GB:

# Run the benchmark script
python scripts/bench_rtx4060.py

Tests GPT-2 FP16, TinyLlama 1.1B FP16, Qwen2.5-7B NF4 (~5 GB), and GGUF via llama.cpp.

Local coding assistant

Run a coding agent (Aider, Cline, Continue) on a local model, on mismatched consumer GPUs, with OpenAI-compatible tool calling — validated end-to-end (a real Aider session edits code and updates tests, unassisted):

./serve_qwen36.sh                              # Qwen3.6-35B-A3B on 2 GPUs, API on :5030

pip install aider-chat
OPENAI_API_BASE=http://localhost:5030/v1 OPENAI_API_KEY=dummy \
  aider --model openai/qwen3.6-coder --no-stream

Function calling works (parses Qwen's <tool_call> format → OpenAI tool_calls), the full tool round-trip is handled, and malformed calls never escape. Full guide, other agents (Cline/Continue), pitfalls and measured perf: docs/coding_agents.md.

Perf note: the coding path runs through llama.cpp/GGUF. The prompt-lookup +500% number was measured on the HuggingFace backend, not this path — see BENCHMARK_RESULTS.md.

Usage

One-command inference (recommended)

# Interactive mode — loads model, then prompts you
vramancer run Qwen/Qwen2.5-7B-Instruct

# One-shot with prompt
vramancer run mistralai/Mistral-7B-v0.1 -p "What is VRAM?" --max-tokens 128

# With quantization (nf4, int8, nvfp4 for Blackwell)
vramancer run Qwen/Qwen2.5-14B-Instruct -q nf4

# Force specific GPU count
vramancer run meta-llama/Llama-3-8B --gpus 2

# GGUF models auto-select llama.cpp (faster than HF for GGUF: ~106 vs ~35 tok/s on a 7B here)
vramancer run bartowski/Qwen2.5-7B-Instruct-GGUF

API server (OpenAI-compatible)

# Start server with model pre-loaded
vramancer serve --model Qwen/Qwen2.5-7B-Instruct --port 5030

# Then query it
curl http://localhost:5030/v1/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $VRM_API_TOKEN" \
  -d '{"prompt": "Hello", "max_tokens": 64}'

Other commands

vramancer doctor      # Full diagnostic (GPU, P2P, versions, health) — measured numbers only
vramancer status      # Show GPUs, memory, backend
vramancer health      # System health check
vramancer dashboard   # Real-time web dashboard (GPU/VRAM/tok-s)
vramancer history     # Recent requests (tok/s, OOM, trends)
vramancer hub Qwen/Qwen2.5-14B-Instruct  # Browse model formats on HF
vramancer benchmark   # GPU matmul benchmark
vramancer split Qwen/Qwen2.5-14B-Instruct --gpus 2  # Preview model split

Cluster (data-parallel across GPUs)

# Data-parallel: one worker process per GPU, requests routed by work-stealing (~2× on 2 GPUs).
vramancer cluster serve Qwen/Qwen2.5-14B-Instruct   # OpenAI API on :5040, dashboard on :5041/dash

Auto-restarts dead workers, records history, alerts on failure (VRM_ALERT_WEBHOOK).

Add another machine to the cluster (multi-node)

Each machine joins with one command — the installer pulls inference + mDNS by default, so the node auto-announces on the LAN. Works across backends (a Mac MPS node + a CUDA node cooperate; the gateway only speaks HTTP).

# 1. On the new machine (laptop / Mac / desktop) — one command:
curl -fsSL https://raw.githubusercontent.com/thebloodlust/VRAMancer/main/install.sh | bash
#    (Windows: irm https://raw.githubusercontent.com/thebloodlust/VRAMancer/main/install.ps1 | iex)

# 2. Run a node on that machine (it announces itself via mDNS):
vramancer serve Qwen/Qwen2.5-7B-Instruct --port 5040

# 3. On any machine, start the gateway — it discovers nodes and routes whole requests:
vramancer cluster gateway --discover
#    or explicitly: vramancer cluster gateway --nodes http://laptop.local:5040,http://mac.local:5040

Not a GUI "1-click" (it's one terminal command): a true single binary can't bundle PyTorch/CUDA + the kernel-level GPU driver. The installer is the honest equivalent.

The same brick is also the foundation for cross-vendor (NVIDIA+AMD in one box) — pending an AMD GPU. See docs/CLUSTER.md.

How it works

1. Auto-detect GPUs — enumerates all CUDA/ROCm/MPS devices with free VRAM
2. Compute-aware memory map — computes a max_memory budget per GPU (favouring the faster GPU) and hands it to accelerate / llama.cpp / vLLM, which do the actual layer placement and dispatch. This avoids the fp32-upcast OOM that the naive 97% formula triggers on tight-fit models.
3. Inference via the chosen engine — accelerate runs the forward pass (pipeline-parallel across GPUs); VRAMancer does not reimplement it.
4. Quantization — optional NF4/INT8/NVFP4 to reduce VRAM footprint
5. KV cache management — paged attention with optional PolarQuant+QJL compression (~3.5 bits/dim, ~4.6x reduction)

Honest scope: VRAMancer's value is the orchestration, the OOM-avoiding placement, the measured optimisations and the UX (one-command install, quickstart, doctor, dashboard, mDNS cluster) — not a from-scratch inference engine. Weight-tiering, MoE expert streaming and prefill/decode disaggregation were prototyped and measured to not beat the standard engines on this hardware; they are not claimed as features.

Backends

Tip: GGUF models are auto-detected and use llama.cpp automatically. For HuggingFace models, add --backend llamacpp isn't needed — just use a GGUF repo name.

Compatibility matrix

Legend: ✅ supported & tested · ⚠️ partial / experimental · ❌ not supported · N/A not applicable.

BnB (NF4/INT8) multi-GPU has an upstream bug in accelerate 1.13 + transformers 5.3 — VRAMancer forces single-GPU for BnB. Use NVFP4 or GGUF Q4_K_M for multi-GPU quantized inference.

Configuration

VRAMancer is configured via environment variables (VRM_*), not config files:

VRM_QUANTIZATION=nf4          # Quantization mode
VRM_KV_COMPRESSION=turboquant # KV cache compression (~4.6x reduction)
VRM_PARALLEL_MODE=pp           # pp (pipeline) or tp (tensor parallel)
VRM_API_TOKEN=your-token       # API authentication
VRM_PRODUCTION=1               # Strict security mode

Full list: .github/copilot-instructions.md

Development

# Stub tests (no GPU required)
VRM_MINIMAL_TEST=1 VRM_DISABLE_RATE_LIMIT=1 pytest tests/ -m "not gpu and not real_torch and not heavy and not chaos" -q

# GPU integration tests (requires CUDA/ROCm/MPS)
VRM_MINIMAL_TEST=1 VRM_DISABLE_RATE_LIMIT=1 pytest tests/ -m "gpu or real_torch" -q

# Lint
flake8 core/ tests/

Architecture

vramancer run model
    └─ InferencePipeline
        ├─ backends.select_backend()  → HuggingFace / vLLM / Ollama / llama.cpp
        ├─ model_splitter             → VRAM-proportional layer assignment
        ├─ TransferManager            → GPU-to-GPU data movement (P2P or CPU-staged)
        ├─ StreamManager              → prefetch, swap, eviction
        └─ continuous_batcher         → batched inference (optional)

Detailed architecture: docs/architecture.md

Known limitations & technical debt

See docs/reports/TECHNICAL_DEBT.md for documented stubs, known limitations (BnB multi-GPU upstream bug, CUDA Graph single-GPU only, etc.) and V4 plan outcomes.

License

MIT