memra

Rust + CUDA inference engine tuned separately for RTX PRO 6000 Blackwell and RTX 5090, with OpenAI-compatible serving and model-specific correctness gates.

Install · Models · Serving · Performance · Hosted API

Want to try Memra without operating a GPU? Open the hosted instance →. Its live model catalog, access path, prices, and terms are documented there.

Quick start

Prebuilt binaries require Linux x86_64, an NVIDIA driver with CUDA 13 runtime support, and the CUDA runtime libraries. The installer selects the GPU build, verifies its checksum, and does not require nvcc.

curl -fsSL https://raw.githubusercontent.com/avifenesh/memra/main/tools/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
kernel-check                  # expect: ALL GREEN

Start a server with a supported public checkpoint. The first run downloads and caches the model:

MEMRA_MODELS="q38=hf:Avifenesh/Qwen3.8-27B-NVFP4-MTP-GGUF:Q5K-mtp" \
memra-server

The server listens on 127.0.0.1:8080. Send a streaming chat completion from another terminal:

curl -sS -N http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "q38",
    "messages": [{"role": "user", "content": "Explain KV caching in one sentence."}],
    "max_tokens": 128,
    "stream": true
  }'

Use the cookbook for qualified model-and-card configurations. See installation for source builds and platform requirements.

Project spirit

Memra is narrow on purpose. The best execution path depends on the model, checkpoint, hardware, and workload, so Memra measures those combinations separately instead of promising that one loader or one default supports everything. A fast path is promoted only when its own correctness and performance evidence is current.

The project favors explicit support, reproducible receipts, and useful failures over silent fallbacks. Issues, model requests, hardware reports, and evidence-backed pull requests are welcome. Look elsewhere if you need broad hardware coverage, a large architecture catalog, or a Python library embedded in your application.

Choose a path

By model: Qwen3.8 27B · Qwen3.5 9B · Qwen3.6 27B · Qwen3.6 35B-A3B · Qwen-AgentWorld 35B-A3B · Ornith 1.5 35B-A3B · Ornith 1.0 9B · Ornith 1.0 35B · Gemma 4 12B · Gemma 4 26B-A4B · Gemma 4 31B · Gemma 4 E4B · Step 3.7 Flash · DeepSeek V4 Flash

By hardware: RTX PRO 6000 Blackwell · RTX 5090 / 50-series · H100 · Ada · B200

By workload: interactive agents · long-context and prefix reuse · batch throughput · multimodal · large models on multiple GPUs

Documentation

Issues and requests

Issues and requests are welcome:

• Report a bug
• Request a model or feature
• Submit a hardware validation
• Report a vulnerability privately

For a model request, include the exact checkpoint, quantization, target GPU, and why the model is useful. For a performance report, include the command, model artifact, hardware, and raw output.

Contributing

Issues are welcome even without a proposed fix. Pull requests are welcome when they include the proof required for the affected path; GPU changes cannot be validated by GitHub's compile-only runners. Start with CONTRIBUTING.md.

License

MIT — see LICENSE. Built by Avi Fenesh at tiyuvta.