Compile → Benchmark → Deploy. Emmy compiles PyTorch models into CUDA kernels, benchmarks LLM inference, and deploys models on local or cloud GPUs with vLLM and SGLang.

Compile · Tune · Deploy · Benchmark · Serve · Recipes

See Emmy in action: Outperforming vLLM on Gemma4-12B.

Install

Requires Python 3.12 or newer.

pip install emmy-ml
emmy --version

The package includes ready-to-use recipes. Remote deployment and benchmarking do not require a local GPU or CUDA installation. For gated Hugging Face models, accept the model's license and set HF_TOKEN.

Compile

Compile a PyTorch operation, compare its speed with PyTorch, or inspect the generated CUDA:

pip install "emmy-ml[compile]"

# Compile an operation
emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))"

# Benchmark against eager PyTorch and torch.compile
emmy run --bench -c "nn.Softmax(dim=-1)(torch.randn(1,28,2048,2048))"

# Inspect the generated CUDA
emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))" --ir cuda

Running compiled kernels requires an NVIDIA GPU and the CUDA toolkit (nvcc); source builds also need Rust. For GPUs older than Turing, use CUDA 12 and a PyTorch build that supports your GPU. Fast math is enabled by default; set EMMY_FAST_MATH=0 to disable it.

You can also compile a custom module:

emmy compile -c "
class LN(nn.Module):
    def forward(self, x):
        m = x.mean(-1, keepdim=True)
        v = ((x - m) ** 2).mean(-1, keepdim=True)
        return (x - m) * torch.rsqrt(v + 1e-6)
LN()(torch.randn(64, 2048))"

For Python integration, use torch.compile(model, backend="emmy", dynamic=False) under torch.no_grad(). Read the compiler guide or the compiler walkthrough for more.

Lowered kernel example

Readable Schedule: emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))" --ir tile

kernel k_rms_norm_reduce  inputs: rms_norm_mean_count, rms_norm_eps, x, p_weight  outputs: rms_norm
    in0 = load rms_norm_mean_count[0]
    in1 = load rms_norm_eps[0]
    Tile(axes=(a0:256=THREAD, a1:32=BLOCK)):
        x_smem = Stage(x, origin=(0, a1, 0), slab=(a2:2048@2)) async
        p_weight_smem = Stage(p_weight, origin=(0), slab=(a3:2048@0)) async
        StridedLoop(a2 = a0; < 2048; += 256):  # reduce
            in2 = load x_smem[a2]
            v0 = multiply(in2, in2)
            acc0 <- add(acc0, v0)
        v1 = divide(acc0, in0)
        v2 = add(v1, in1)
        v3 = rsqrt(v2)
        StridedLoop(a3 = a0; < 2048; += 256):  # free
            in3 = load x_smem[a3]
            in4 = load p_weight_smem[a3]
            v4 = multiply(in3, v3)
            v5 = multiply(v4, in4)
            rms_norm[0, a1, a3] = v5

Optimized CUDA kernel: emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))" --ir cuda

extern "C" __global__
__launch_bounds__(256) void k_rms_norm_reduce(const float* x, const float* p_weight, float* rms_norm) {
    float in0 = 2048.0f;
    float in1 = 1e-06f;
    {
        int a1 = blockIdx.x;
        int a0 = threadIdx.x;
        float acc0 = 0.0f;
        __syncthreads();
        __shared__ float x_smem[2048];
        for (int x_smem_flat = a0; x_smem_flat < 2048; x_smem_flat += 256) {
            {
                unsigned int _smem_addr = __cvta_generic_to_shared(&x_smem[x_smem_flat]);
                asm volatile("cp.async.ca.shared.global [%0], [%1], 4;\n"
                             :: "r"(_smem_addr), "l"(&x[a1 * 2048 + x_smem_flat])
                             : "memory");
            }
        }
        asm volatile("cp.async.commit_group;\n" ::: "memory");
        asm volatile("cp.async.wait_group 0;\n" ::: "memory");
        __syncthreads();
        __shared__ float p_weight_smem[2048];
        for (int p_weight_smem_flat = a0; p_weight_smem_flat < 2048; p_weight_smem_flat += 256) {
            {
                unsigned int _smem_addr = __cvta_generic_to_shared(&p_weight_smem[p_weight_smem_flat]);
                asm volatile("cp.async.ca.shared.global [%0], [%1], 4;\n"
                             :: "r"(_smem_addr), "l"(&p_weight[p_weight_smem_flat])
                             : "memory");
            }
        }
        asm volatile("cp.async.commit_group;\n" ::: "memory");
        asm volatile("cp.async.wait_group 0;\n" ::: "memory");
        __syncthreads();
        for (int a2 = a0; a2 < 2048; a2 += 256) {
            float in2 = x_smem[a2];
            float v0 = in2 * in2;
            acc0 += v0;
        }
        __shared__ float acc0_smem[256];
        acc0_smem[a0] = acc0;
        __syncthreads();
        for (int s = 128; s > 0; s >>= 1) {
            if (a0 < s) {
                acc0_smem[a0] = acc0_smem[a0] + acc0_smem[a0 + s];
            }
            __syncthreads();
        }
        __syncthreads();
        float acc0_b = acc0_smem[0];
        float v1 = acc0_b / in0;
        float v2 = v1 + in1;
        float v3 = rsqrtf(v2);
        for (int a3 = a0; a3 < 2048; a3 += 256) {
            float in3 = x_smem[a3];
            float in4 = p_weight_smem[a3];
            float v4 = in3 * v3;
            float v5 = v4 * in4;
            rms_norm[a1 * 2048 + a3] = v5;
        }
    }
}

Tune

Try 20 schedules for a single kernel on your GPU, with cold-cache measurements:

emmy run --bench --cold-cache --tune 20 -c "nn.Linear(4096,4096,bias=False)(torch.randn(1,4096))"

Successful measurements are saved so later compiles can reuse them when the kernel and measurement settings match.

Fit the priors

The schedule prior ranks kernel schedules; the placement prior chooses how to split work into kernels. Both guide compilation when no compatible measurement is available. You can refit them from the repository's golden files in a source checkout, without a GPU:

# Load the repository goldens into a separate database
emmy db import --db _data/dataset.db --fresh --repository

# Export a dataset for each prior
emmy db export --db _data/dataset.db _data/schedule --space schedule
emmy db export --db _data/dataset.db _data/placement --space placement --repository

# Refit the shipped weights
emmy fit _data/schedule emmy/compiler/pipeline/search/prior/weights/schedule.json
emmy fit _data/placement emmy/compiler/pipeline/search/prior/weights/placement.json

# Evaluate both priors
emmy eval prior _data/schedule
emmy eval prior _data/placement

To experiment without replacing the shipped weights, fit a separate candidate and compare it on the same dataset:

emmy fit _data/schedule _data/schedule-candidate.json --folds 0
emmy eval prior _data/schedule \
  --offline-file emmy/compiler/pipeline/search/prior/weights/schedule.json \
  --compare-to _data/schedule-candidate.json --json _data/schedule-comparison.json

Adjust --iterations, --depth, or --learning-rate to try different fits. --folds 0 skips cross-validation; omit it to use the default five folds. Each fit writes metrics under _tune/fits/, and the comparison command writes a JSON report. See the CLI reference for all tuning and fitting options.

Deploy

Choose a recipe that matches your GPU. This Gemma recipe targets one RTX 5090:

emmy recipe list

# Local Docker Compose
emmy deploy local --recipe gemma-4-12B-it

# An existing server
emmy deploy ssh --recipe gemma-4-12B-it --ssh user@host

# Rent a cloud VM and deploy
emmy deploy cloud --recipe gemma-4-12B-it

A recipe name creates an editable copy in the current directory. You can also pass a recipe directory with --recipe path/to/recipe. Deployment leaves the server running and prints its endpoint.

Local deployment needs Docker Compose and NVIDIA container support. Cloud deployment needs provider credentials: CLOUDRIFT_API_KEY for CloudRift, or an authenticated gcloud CLI for GCP. See the deployment reference for teardown, provider options, and multi-model plans.

Benchmark

From a source checkout, run an experiment on cloud GPUs, your machine, or existing servers:

# Rent GPUs, run one experiment, and tear down
emmy bench experiments/gemma-4-12B/gsm8k_mtp_rtx5090

# Select RTX 5090 variants across Gemma experiments
emmy bench experiments/gemma-4-12B/* --filter "deploy.gpu=*5090*"

# Use this machine or an existing server
emmy bench experiments/gemma-4-12B/gsm8k_mtp_rtx5090 --local
emmy bench experiments/gemma-4-12B/gsm8k_mtp_rtx5090 --ssh user@host

Results and logs are saved in a timestamped directory beside the experiment recipe. Cloud VMs are removed after the run; existing hosts are kept. Local benchmarks use SSH to localhost, so local SSH access must be configured. Browse experiments and results or read the benchmark reference.

Serve

Serve an embedding model through vLLM with Emmy-compiled kernels:

pip install "emmy-ml[compile,serving]"
emmy serve Qwen/Qwen3-Embedding-0.6B

Then send a request from another terminal:

curl localhost:8000/v1/embeddings -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":"Hello"}'

For a one-shot benchmark and a stock-vLLM comparison:

emmy serve Qwen/Qwen3-Embedding-0.6B --bench --random-input-len 32
emmy serve Qwen/Qwen3-Embedding-0.6B --bench --random-input-len 32 --stock

See the serving guide for text generation and the native generation guide for the experimental Rust server.

Recipes

A recipe sets the model, serving engine, GPU, and optional benchmark workload. Save this as my-benchmark/recipe.yaml and run emmy bench my-benchmark:

model:
  huggingface: "Qwen/Qwen3-0.6B"

engine:
  llm:
    tensor_parallel_size: 1
    context_length: 4096
    vllm:
      image: "vllm/vllm-openai:v0.23.0"

benchmark:
  num_prompts: 256
  random_input_len: 1024
  random_output_len: 128

# Cross-product: 2 GPUs × 2 concurrency settings = 4 variants
matrices:
  cross:
    deploy.gpu_count: 1
    deploy.gpu: ["NVIDIA GeForce RTX 5090", "NVIDIA H100 80GB"]
    zip:
      engine.llm.max_concurrent_requests: [32, 128]
      benchmark.max_concurrency: [32, 128]

Browse serving recipes for existing configurations. The recipe reference covers fields, parameter sweeps, LoRA adapters, and custom commands.

Virtual Machine Management

# Rent a GPU server and save its lease for cleanup
emmy vm create gpu --gpu "NVIDIA GeForce RTX 5090" --gpu-count 1 --exact-gpu-count \
  --lease /tmp/emmy-vm.json --owner local-run --json

# Delete that server
emmy vm delete lease /tmp/emmy-vm.json --owner local-run

See the VM reference for provider-specific commands and availability queries.

Documentation

• CLI reference — command options, tuning, refitting priors, and image publication
• Compiler tutorials — tracing, optimization, and measuring kernels
• Glossary — compiler and serving terminology

Related Projects

These repositories provide the platform, serving runtimes, and deployment services that Emmy integrates with. The CloudRift repositories are typically checked out beside Emmy, so use ../<repository-name> for local code inspection when that sibling exists, and verify the remote source when current compatibility matters. Some are internal and require CloudRift organization access.

Project Structure

• AGENTS.md — shared coding-agent instructions; CLAUDE.md imports the same guidance for Claude Code
• .agents/skills/ — canonical repository skills; .claude/skills/ exposes them through compatibility symlinks
• prompts/ — reusable agent prompts shared by repository skills and non-interactive workflows
• .opencode/ — API-agent provider, permissions, and workflow profiles
• .github/ — Pull-request checks, releases, cloud experiments, and model discovery/onboarding workflows (see ARCHITECTURE.md)
• emmy/ — Python package
  • emmy.py — CLI entrypoint
  • logging_setup.py — CLI logging configuration
  • hardware.py — GPU specs and instance type mapping
  • detect.py — GPU detection via PCI sysfs (local and remote)
  • redact.py — Secret redaction for logs and dumps
  • commands/ — CLI layer (thin argparse handlers, see ARCHITECTURE.md)
    • deploy/ — deploy local, deploy ssh, deploy cloud commands
    • bench/ — bench command
    • vm/ — vm create/delete/audit commands (GCP, CloudRift, owned leases)
    • teardown.py — teardown command
    • pull.py — pull command (download HF model)
    • trace.py — trace command (PyTorch → Graph IR)
    • compile.py — compile command (decomposition → optimization → fusion → kernel/CUDA lowering)
    • run.py — run command (compile + execute on CUDA backend, optional benchmarks)
    • inspect_graph.py — inspect command (graph summary)
  • compiler/ — PyTorch → Graph IR → CUDA compiler (see ARCHITECTURE.md)
    • graph.py — Graph, Node, Tensor, Hints container
    • pretty.py — graph dumps as Rust-shaped pseudocode (see IR-PSEUDOCODE-TORCH.md)
    • ir/ — per-dialect op definitions (torch / tensor / loop / kernel / cuda) (see ARCHITECTURE.md)
      • schedule/ — schedule interfaces, reusable choices, and classic scheduling (see ARCHITECTURE.md)
    • trace/ — PyTorch/HuggingFace → Graph IR capture (see ARCHITECTURE.md)
    • pipeline/ — rewrite engine + passes + dump hooks (see ARCHITECTURE.md)
    • backend/ — numpy / loop / CUDA execution (see ARCHITECTURE.md)
      • cuda/ — CUDA backend internals (see ARCHITECTURE.md)
  • recipe/ — Recipe loading, dataclass types, engine flag mapping (see ARCHITECTURE.md)
  • serving/ — vLLM out-of-tree embedding plugin (see ARCHITECTURE.md)
  • deploy/ — Compose generation, deploy orchestration
  • provisioning/ — Cloud provisioning, SSH transport, VM lifecycle
  • benchmark/ — Benchmark tracking, config, task enumeration, execution (see ARCHITECTURE.md)
  • planner/ — Groups benchmark tasks into execution groups for VM allocation
• recipes/ — The recommended serving configuration and per-GPU model goldens, one directory per model — what emmy deploy runs (see ARCHITECTURE.md; benchmark grids belong in experiments/)
• docker/ — Custom image builds (vllm-emmy — vLLM + the emmy plugin; vllm-emmy-serve — prebuilt per-model images: warmed cubins + baked model snapshot; 1cat-vllm-sm70 — source-pinned 1Cat-vLLM runtimes and request-time GPU caches for Volta)
• experiments/ — Benchmark parameter sweeps, self-contained recipe + committed results — what emmy bench runs (see ARCHITECTURE.md)
• kernels/ — Standalone CUDA kernel sources
• crates/emmy-runtime/ — The Rust runtime: the executor behind every launch, the standalone pack loader and cached generation (design, hosts, and qualification in ARCHITECTURE.md); crates/emmy-runtime-py/ is its in-process host, the emmy.emmy_runtime extension setuptools-rust builds into the package
• crates/emmy-server/ — Native text and HTTP adapter (see ARCHITECTURE.md)
• docs/ — Docusaurus user-docs site (getting started, benchmarking, custom configurations, deployment)
• tests/ — pytest tests (see ARCHITECTURE.md)
  • compiler/passes/ — compiler pass tests (see ARCHITECTURE.md)
  • compiler/realization/ — the realization corpus: checked-in reproducers of pinned schedules, replayed as data (see ARCHITECTURE.md)
  • perf/ — GPU perf comparison vs PyTorch (see ARCHITECTURE.md)
• scripts/ — Analysis and visualization scripts
• utils/ — Standalone utility scripts
• config.yaml — Benchmark configuration
• Makefile — Build automation
• pyproject.toml — Package metadata and tool config

Contributing

Source setup requires Python 3.12+ and a Rust toolchain:

git clone https://github.com/cloudrift-ai/emmy.git
cd emmy
make setup

Follow STYLE.md and the contribution instructions for development, testing, and PRs. Release procedures live in the workflow documentation.

License

Licensed under the Apache License 2.0.