hipEngine

hipEngine is a ROCm-native local inference engine built primarily for AMD Radeon GPUs. It pairs a small Python host with custom HIP kernels for torch-free model loading, generation, and OpenAI-compatible serving on supported hardware.

Current release: v0.5.0. Besides the Qwen 3.6 PARO and GGUF models, the latest version of hipEngine now supports GGUF inference for more model families. These include Laguna S 2.1, Maple ternary, and Moonshine ASR.

Why use hipEngine?

• Native AMD support. HIP-first kernels directly target and tune for specific RDNA 3 (gfx1100) and Strix Halo RDNA 3.5 (gfx1151) instead of being CUDA ports.
• No PyTorch runtime required. There is no PyTorch dependency, which keeps hipEngine lightweight. Although it is packaged for Python, almost all of the hot path is C++.
• Optimized for agents and concurrent requests. Besides extensive tuning for fast single-request performance (especially for prefill), hipEngine also has tuned support for c=N. It is significantly faster than llama.cpp or vLLM for c=8 workloads.
• Drop-in support for existing clients. The included OpenAI-compatible server supports completion, chat, token-level SSE, logprobs, tools, structured-output validation, Qwen thinking controls, logprob-biased effort control, and request diagnostics.

hipEngine is a new, small software project focused on making a select list of models perform well, particularly Qwen 3.x variants and fine-tunes.

Supported models

Yes means that public text generation has been tested. A dash means that the combination is not supported. The Qwen rows group closely related model versions, with size-specific format coverage shown explicitly. Features such as batching, sampling, tools, and long context can differ by model.

CPU model generation is not supported. The CPU backend is used for correctness tests. On NVIDIA, load Maple with backend="cuda_sm120a"; automatic hardware selection currently covers AMD only.

Support is specific to the listed model families and formats. hipEngine does not yet run every GGUF model. See the GGUF, Laguna, and Maple guides for model-specific limits.

GGUF or ParoQuant for Qwen?

For Qwen3.6 35B-A3B on W7900, the optimized ParoQuant W4 checkpoint currently leads short-context generation and uses less memory. GGUF leads prompt processing from 1K tokens onward in the current six-shape sweep.

GGUF has a much larger model and quantization ecosystem. Current development is therefore focused on GGUF compatibility. Choose PARO for this exact optimized checkpoint or GGUF for broader compatibility.

Installation

Requirements

ROCm 7.x is the safest choice for the current wheel. The first model load compiles and caches kernels, so it takes longer than later starts.

Install from PyPI:

pip install hipengine huggingface_hub

Or install a source checkout:

git clone https://github.com/shisa-ai/hipEngine.git
cd hipEngine
git lfs install
git lfs pull
pip install -e .

Confirm that the command is available:

hipengine --help
hipengine serve --help

Start a local server

hipEngine does not download model weights during startup. Download a supported model first, or use a GGUF file that is already on disk.

For the ParoQuant Qwen checkpoint:

hf download shisa-ai/Qwen3.6-35B-A3B-PARO-packed

hipengine serve \
  --model shisa-ai/Qwen3.6-35B-A3B-PARO-packed \
  --served-model-name qwen-paro

For GGUF, pass the path to the model file:

hipengine serve \
  --model /path/to/Qwen3.6-35B-A3B-Q4_K_M.gguf \
  --served-model-name qwen

The server listens on http://127.0.0.1:8000 by default. Test it with:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen",
    "messages": [{"role": "user", "content": "Why is the sky blue?"}],
    "max_tokens": 128
  }'

Point any client that accepts a custom OpenAI base URL at http://127.0.0.1:8000/v1. See the server guide for API keys, streaming, tools, structured output, and model capability checks.

Use the Python API

from hipengine import LLM, SamplingParams

llm = LLM("shisa-ai/Qwen3.6-35B-A3B-PARO-packed")
outputs = llm.generate(
    ["Hello, hipEngine."],
    SamplingParams(max_tokens=64, temperature=0.0),
)
print(outputs[0])
llm.close()

LLM(...) detects a supported AMD GPU and chooses the model format automatically. You can also pass a local GGUF or Maple path. Advanced users can override the choice with backend= and quant=. The opt-in execution_profile="strict"|"production"|"batch_invariant" selector is fail-closed to registered kernel plans with exact fallbacks; omitting it preserves the migration default until profile calibration completes.

Performance highlights

These are measured results, not estimates. Prompt processing is the speed of reading the input. Text generation is the speed of producing new tokens.

Every number below is measured on the named hardware and links to a reproducible artifact. Prompt processing is how fast hipEngine reads your input; text generation is how fast it writes new tokens. With MTP is speculative decoding, which is enabled only where it is qualified for that model and shape. Rows use different models and protocols — compare within a row, not across them.

Performance

Radeon Pro W7900 — 48 GB (gfx1100)

Strix Halo / Radeon 8060S — 120 GB (gfx1151)

NVIDIA RTX PRO 6000 Blackwell — 96 GB (sm_120a)

Serving several requests at once

hipEngine is very strong at multi-concurrency vs llama.cpp (or even vLLM). Aggregate tokens per second across all active requests, Qwen3.8-27B Q4_K_M on the W7900 under one server protocol; the peers use F16 KV where hipEngine uses BF16.

Direct engine route on the same card and model, 512-token prompts and 128 generated tokens per request, showing what each added request costs in memory:

Eight concurrent requests need about 25 GiB, so this shape wants a 32 GB or larger card. What the same model tolerates on a 24 GB card is not qualified yet.

On Strix Halo, Maple-Preview 2-bit scales to 214.788 tok/s across eight requests (123.131 at one, 165.697 at two, 202.038 at four). Where speculative decoding runs automatically in production it is scoped to a qualified shape: Qwen3.6-35B-A3B GGUF reaches 93.644 tok/s public — 1.1565x its own AR — at two concurrent requests on the W7900.

Full commands, software versions, model hashes, memory use, and correctness checks are in the benchmark report.

Status and limits

v0.5.0 adds the dense Qwen models, and hipEngine now picks some performance routes on its own, but only where they have been measured as safe:

• Qwen3.6-27B and Qwen3.8-27B GGUF generation and serving on both AMD backends. Both can speculate with the model's own multi-token prediction head, as can Qwen3.6-35B-A3B.
• The server turns speculative decoding on only for the model, GPU, and request shapes it has measured, reports why when it skips speculation, and can be switched off for all new requests by one endpoint call.
• An optional execution-profile selector (strict, production, batch_invariant) that runs a registered kernel plan, checks that its fallbacks are installed, and rejects a combination hipEngine has not been shown to complete.
• Scheduling and memory defaults: the fair prefill/decode policy, a smaller per-process GPU memory reserve on Radeon RDNA 3, and FP16 recurrent state for Qwen3.8 Q4_K_S on Strix Halo.
• Still supported: Qwen3.5/3.6 GGUF and ParoQuant, Laguna S 2.1, Maple-Preview, several requests at once on one resident model, and OpenAI-compatible streaming, sampling, tools, structured-output validation, and cancellation.

Full user-facing change history is in the changelog.

Important limits:

• hipEngine uses one GPU. Multi-GPU inference is not implemented.
• There is no desktop GUI, model catalog, or automatic model download.
• CPU model inference is not implemented.
• NVIDIA support is limited to single-request Maple generation through the Python API. CUDA server and multi-request support are not ready.
• Maple currently uses greedy generation only.
• Advertised model context lengths are not a promise that hipEngine supports the same length. Use the model guide and set a conservative server context limit. Repeated 128K context on Strix Halo can still stall, so no 128K number is published.
• The memory figures come from a 48 GB W7900. What Qwen3.8-27B tolerates on a 24 GB card is not qualified yet, so keep a conservative context limit there.
• Automatic speculative decoding covers only narrow measured shapes. Asking for it explicitly on a dense Qwen model uses three draft tokens and can produce different text from normal decoding; HIPENGINE_GGUF_MTP_VERIFY_MODE=serial_exact restores token-for-token agreement. See Server API.
• APIs and supported combinations can still change before 1.0.

Hardware detection

backend="auto" recognizes gfx1100 and gfx1151. These cover the tested Radeon RX 7900 XTX / Pro W7900 and Ryzen AI MAX+ 395 / Radeon 8060S systems. Other AMD architecture numbers are not automatically treated as compatible.

You can force a nearby backend, but do so only after checking output quality and performance. hipEngine will not silently use PyTorch when a GPU is unsupported.

Documentation

User guides

Development and benchmark details

Project lineage

hipEngine is an independent project that builds on ideas and software from ROCm, HIP, Nano-vLLM, ParoQuant, FastDMS, llama.cpp, and other open-source projects. See the source and model guides for detailed attribution.

License

hipEngine source code is licensed under AGPL-3.0-or-later. Model weights, checkpoints, and external datasets remain under their own licenses.