reCUDA

reCUDA is an experimental kernel search and runtime optimization workbench for CUDA LLM workloads.

It is the repo where we try to answer questions like:

• can we generate a faster Triton kernel for this model op?
• does that kernel still win once it is patched into a real runtime path?
• when should we keep Triton, switch to CUTLASS, or just leave cuBLAS alone?

Under the hood, reCUDA mixes retrieval, DSPy orchestration, PEPA / GEPA-style optimization, runtime patching, and a lot of benchmark bookkeeping to search over Triton, CUTLASS, and CUDA-adjacent kernel implementations.

This is not a polished compiler and it is not a universal autotuner. It is a practical, imperfect research system for generating, repairing, validating, benchmarking, and selectively shipping better kernels.

What reCUDA does

• Kernel search: generate candidate kernels for real model ops, usually with Triton.
• Multi-stage optimization: use DSPy, retrieval, and PEPA / GEPA-style loops to improve candidates instead of doing one-shot prompting.
• Runtime patching: patch selected kernels back into real Nemotron and Qwen execution paths.
• Artifact tracking: keep results, prompts, modules, benchmark outputs, and search state on disk.
• Runtime comparison: compare Triton candidates against PyTorch, cuBLAS-backed baselines, and CUTLASS grouped GEMM paths.

If you want the shortest description, reCUDA is a kernel research lab with a CLI.

Current status

• 45 verified Triton kernels across 6 model families. Full tables live in kernels/KERNELS.md.
• Nemotron 3 Nano 30B currently has a bounded, exact-match-safe release preview for the router subset, usually around 1.13x-1.23x overall in the repo-native BF16 decode proxy.
• The narrowed Nano 30B rmsnorm release path is also exact-match-safe on the bounded suite and usually lands around 1.07x-1.21x.
• Nemotron 3 Super, Nemotron 3 Nano / Cascade 2, Nemotron Nano 4B / 9B, and Qwen 3.5 all have meaningful task-level kernel coverage.

The current Nano 30B release framing is here: docs/release_nemotron_nano30b_router.md

Measured highlights

These are the high-signal outcomes, not the full scoreboard.

For raw kernel tables, latencies, and source provenance, see kernels/KERNELS.md.

Quick start

Set up the repo-local environment:

uv venv --system-site-packages .venv
source .venv/bin/activate
uv pip install -e ".[openai,rlm,local-model,dspy,results-db,dev]"
export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas

Check the local stack:

cuda-rlm inspect-env
cuda-rlm presets

If you just want one real demo, start here:

cuda-rlm nano30b-router-release

That is the cleanest "show me a real runtime result" path in the repo right now.

Common workflows

1. Run a release-style benchmark

cuda-rlm nano30b-router-release
cuda-rlm nano30b-rmsnorm-release
cuda-rlm nano30b-best-release \
  --prompt-json artifacts/runtime-smoke/nemotron3-nano-30b-comparison-prompts.json

Important scope:

• this is a real checkpoint-backed runtime path
• this is a repo-native BF16 decode proxy built from the NVFP4 checkpoint tensors
• this is not a blanket claim about official Hugging Face quantized runtime support
• this is not a claim that every Nano 30B patch in the repo is production-safe

2. Search for a new kernel

Single task:

cuda-rlm run \
  --task cuda_rlm.tasks.qwen35_rmsnorm \
  --preset codex-dspy \
  --enable-retrieval \
  --out-dir artifacts/qwen35-rmsnorm-run

PEPA optimization:

cuda-rlm optimize \
  --task cuda_rlm.tasks.nemotron3_relu2 \
  --out-dir artifacts/pepa-run \
  --iterations 1 \
  --children-per-iteration 1

Multi-task campaign:

cuda-rlm autodialectics \
  --task cuda_rlm.tasks.nemotron3_rmsnorm \
  --task cuda_rlm.tasks.nemotron3_rope \
  --task cuda_rlm.tasks.nemotron3_relu2 \
  --num-candidates 5 \
  --repair-attempts 2 \
  --enable-retrieval \
  --out-dir artifacts/super-dialectics

autoresearch still exists as an alias, but autodialectics is the name to prefer now.

What's inside

• cuda-rlm run: one-task kernel search.
• cuda-rlm optimize: PEPA policy evolution over the search loop.
• cuda-rlm autodialectics: multi-task orchestration, scheduling, retrieval refresh, persistence.
• cuda-rlm nano30b-*: packaged release-style runtime benchmarks for the current Nano 30B path.
• cuda-rlm cutlass-bench: grouped GEMM benchmarking for CUTLASS binaries.
• results.duckdb: run outcomes and scoreboard storage.
• artifacts/kernel_index: retrieval index.
• artifacts/dspy/kernel-search-module: DSPy program state.

Search stack

The main search loop is no longer "prompt a model once and save the result."

reCUDA generally works like this:

1. define a task with a reference implementation and benchmark shape
2. retrieve related kernels and prior artifacts
3. generate or repair candidates through DSPy-backed prompting
4. validate correctness against the reference
5. benchmark candidates on the real device
6. optionally evolve the search policy with PEPA / GEPA-style loops
7. patch safe winners back into runtime harnesses when that is worth doing

Current backend guidance:

• codex-dspy is the default path and the main supported search workflow.
• zai-glm5-dspy is the cheaper API-backed fallback.
• qwen-cli-dspy exists for local Qwen CLI exploratory runs.

What reCUDA is not

This part matters because the repo can look more finished than it really is.

• It is not an official NVIDIA runtime.
• It is not a generic Triton compiler for arbitrary models.
• It is not proof that every faster task-level kernel will improve full-model runtime.
• It is not yet a full local HF/modelopt solution for the Nemotron 3 Nano 30B NVFP4 checkpoint.

The current weak spot is also clear from the benchmarks: standalone routed/shared expert projection patches on Nano 30B are not the obvious next public win. cuBLAS is hard to beat there unless the work becomes more fused or more architecture-specific.

Useful commands

# Inspect the local environment
cuda-rlm inspect-env

# List presets
cuda-rlm presets

# Run one task
cuda-rlm run --task cuda_rlm.tasks.qwen35_rmsnorm

# Run a multi-task search campaign
cuda-rlm autodialectics \
  --task cuda_rlm.tasks.nemotron3_rmsnorm \
  --task cuda_rlm.tasks.nemotron3_rope

# Benchmark CUTLASS grouped GEMMs
cuda-rlm cutlass-bench \
  --json-output artifacts/runtime-smoke/cutlass-bench.json \
  --markdown-output artifacts/runtime-smoke/cutlass-bench.md

# Benchmark a full layer against the best known kernel
cuda-rlm benchmark-layer --model qwen35

# Merge scattered DBs back into the main results store
cuda-rlm merge-results

Repo layout

kernels/              Checked-in Triton kernels by model family
src/cuda_rlm/
  cli.py              Main CLI
  pipeline.py         Search, validation, artifact capture
  dspy_backend.py     DSPy-backed generation path
  autoresearch.py     Legacy module name; autodialectics still routes through here
  tasks/              Kernel task definitions
  examples/           Seed candidates and runtime wrappers
  pepa/               PEPA policy evolution
  results_db.py       DuckDB persistence
scripts/              Benchmark and smoke-test helpers
tests/                Unit and runtime-focused coverage
docs/                 Release notes, runtime notes, architecture docs

Local environment

The repo-local environment currently in use is:

• Python 3.12.3
• PyTorch 2.10.0+cu130
• Triton 3.6.0
• CUDA 13.0
• NVIDIA GB10, SM 12.1, about 120 GiB VRAM

Where to look next

• Full project handoff and current investigative history: AGENTS.md
• Kernel scoreboard: kernels/KERNELS.md
• Current Nano 30B release framing: docs/release_nemotron_nano30b_router.md