ember-iree

IREE-first Qwen runtime scaffold with a working local Qwen3-0.6B path, batch/session contracts for the next serving phase, and the export/validate tooling needed to keep the project moving without another architectural reset.

Current shape

• Qwen3 dense models are the immediate target.
• Runtime is IREE-only from the CLI entry point down.
• prefill and decode_step are wired for single-request execution today.
• Slot lifecycle, reset_slots, and batched contracts are already present in the core API for the next phase.
• Validation and export scripts live in scripts/.
• Qwen3.5 multimodal model configs are recognized. The repo now has working IREE multimodal CUDA paths based on prepared tensor bundles, multistage exporters, and worker-based multi-GPU execution. HF scripts remain the reference path.

For the detailed route review, pitfalls, lessons learned, and the current recommendation on where IREE still makes sense, see:

• docs/iree-retrospective.md
• docs/iree-retrospective.zh.md

Build

The default flow expects an IREE source tree.

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DEMBER_BUILD_TESTS=ON \
  -DEMBER_BUILD_BENCHMARKS=ON \
  -DIREE_SOURCE_DIR=/home/dong/workspace/iree

cmake --build build -j"$(nproc)"

If you do not want the IREE subproject build, set both:

• IREE_RUNTIME_INCLUDE_DIRS
• IREE_RUNTIME_LIBRARIES

Run

CLI:

./build/ember \
  -m /path/to/model_snapshot \
  --artifact-manifest /path/to/manifest.json \
  --driver local-task \
  --prompt "Hello, my name is"

Validation helper:

./build/ember_validate \
  -m /path/to/model_snapshot \
  --artifact-manifest /path/to/manifest.json \
  --driver local-task \
  --prompt "Hello, my name is"

Qwen3 workflow

Run the raw HF reference:

scripts/run_qwen3_hf.sh --prompt "Hello, my name is" --max-new-tokens 16

Export the local Qwen3-0.6B stateless artifact:

scripts/build_qwen3_0_6b_artifact.sh

Run the exported IREE artifact:

scripts/run_qwen3_iree_local_task.sh "Hello, my name is"

Compare HF and ember-iree outputs:

python3 scripts/verify_output.py \
  --model-dir /path/to/model_snapshot \
  --artifact-manifest /path/to/manifest.json \
  --prompt "Hello, my name is"

Mac (Apple Silicon) Support

On Mac, you can run Qwen3.5 models locally with MPS acceleration using the provided scripts.

Setup

1. Create a virtual environment with uv and install dependencies:

uv venv .venv --python 3.14
source .venv/bin/activate
uv pip install torch torchvision torchaudio transformers huggingface_hub accelerate "httpx[socks]"

Run Qwen3.5-0.8B / 2B

Use the Mac-native script to run inference:

# Greely run with default model (Qwen3.5-0.8B)
./scripts/run_qwen3_5_mac.sh --prompt "Explain quantum computing in one sentence."

# Interactive Chat Mode (Streaming)
./scripts/run_qwen3_5_mac.sh --chat

# Run with Qwen3.5-2B (Shorthand)
./scripts/run_qwen3_5_2b_mac.sh --chat

# Or use environment variable to change default
export QWEN_MODEL_ID="Qwen/Qwen3.5-2B"
./scripts/run_qwen3_5_mac.sh --chat

# Options
./scripts/run_qwen3_5_mac.sh --help

Download Models

Models are automatically downloaded, but you can pre-fetch them:

./scripts/download_qwen3_5_mac.sh Qwen3.5-0.8B

Qwen3.5 multimodal reference path

Official Qwen3.5 models support chat + vision. Ember now has two multimodal paths:

1. HF reference runner for direct local validation:

scripts/run_qwen3_5_hf_multimodal.sh \
  --prompt "Describe the image" \
  --image /path/to/example.png

Video input is also wired on the HF path:

scripts/run_qwen3_5_hf_multimodal.sh \
  --prompt "Summarize the video" \
  --video /path/to/example.mp4

Note: video inputs depend on the video decoding backend available in your current transformers environment.

2. Experimental IREE multimodal path:

Build a CUDA multistage artifact from an example image or video:

EXAMPLE_IMAGE=/path/to/example.png \
scripts/build_qwen3_5_multimodal_artifact.sh Qwen3.5-0.8B

For larger multimodal prompts, the build wrapper also accepts newline-delimited EXAMPLE_IMAGE_PATHS / EXAMPLE_VIDEO_PATHS so the exported artifact can be sized from a representative multi-image or mixed image+video example.

Video export is also wired:

EXAMPLE_VIDEO=/path/to/example.mp4 \
scripts/build_qwen3_5_4b_multimodal_artifact.sh

If a local snapshot only contains weights but not processor assets, the build wrapper can point at a compatible processor source explicitly:

EXAMPLE_IMAGE=/path/to/example.png \
PROCESSOR_MODEL_NAME=Qwen3.5-4B \
scripts/build_qwen3_5_35b_a3b_fp8_multimodal_artifact.sh

To inspect whether a local snapshot is actually complete before exporting, run:

python3 scripts/check_qwen_snapshot.py /path/to/model_or_snapshot

That checker also supports machine-readable output and fail-fast modes:

python3 scripts/check_qwen_snapshot.py \
  --format json \
  --fail-on any \
  /path/to/model_or_snapshot

To preview a 35B heterogeneous placement that matches the current machine before exporting, run:

python3 scripts/plan_qwen35_hetero_layout.py \
  --model-name Qwen3.5-35B-A3B-FP8 \
  --snapshot /path/to/model_or_snapshot

The 35B build wrappers now call this preflight planner automatically. It fills in default STAGE_DEVICE_URIS / STAGE_TARGET_BACKENDS from local GPU and RAM capacity and prints any obvious resource warnings up front. Set AUTO_STAGE_PLAN=0 to disable that behavior and manage placement manually. If you only want the preflight output without preparing tensors or exporting, set PLAN_ONLY=1. To treat planner warnings as hard errors, set STRICT_PREFLIGHT=1.

The multistage build wrappers now also expose a real heterogenous placement path. STAGE_TARGET_BACKENDS can be inferred directly from STAGE_DEVICE_URIS, so mixed GPU + CPU placement no longer requires keeping two parallel stage lists in sync.

On this specific machine, a more realistic starting point is the FP8 A3B model:

EXAMPLE_IMAGE=/path/to/example.png \
scripts/build_qwen3_5_35b_a3b_fp8_multimodal_artifact.sh

That wrapper defaults to 10 stages, with the first 4 on the two GPUs and the remaining 6 on CPU. The same placement can also be expressed explicitly:

EXAMPLE_IMAGE=/path/to/example.png \
STAGE_DEVICE_URIS=cuda://0,cuda://1,cuda://0,cuda://1,local-task,local-task,local-task,local-task,local-task,local-task \
scripts/build_qwen3_5_multimodal_artifact.sh Qwen3.5-35B-A3B-FP8

The non-quantized A3B model also has a wrapper:

EXAMPLE_IMAGE=/path/to/example.png \
scripts/build_qwen3_5_35b_a3b_multimodal_artifact.sh

Matching worker launch wrappers also exist for the 35B paths and automatically reuse the Qwen3.5-4B processor assets when the 35B snapshot only contains weights:

MANIFEST=/path/to/manifest.json \
IMAGE_PATH=/path/to/example.png \
scripts/run_qwen3_5_35b_a3b_fp8_multimodal_cuda_workers.sh

There is also a one-shot probe wrapper that chains check -> plan -> build -> run. It defaults to MODE=preflight, so it is safe to use before the local snapshot is complete:

scripts/probe_qwen3_5_35b_a3b_fp8_multimodal.sh

Useful overrides:

• MODE=preflight|build|run|all
• STRICT_PREFLIGHT=1
• CHECK_FAIL_ON=checkpoint|processor|any

But the practical expectation on this box is:

• Qwen3.5-35B-A3B-FP8 is a plausible heterogeneous CPU+GPU experiment
• non-quantized Qwen3.5-35B-A3B is much less realistic here and should be treated as an exploratory path, not the default plan

Run multimodal CUDA workers directly from a fresh image or video:

MANIFEST=/path/to/manifest.json \
IMAGE_PATH=/path/to/example.png \
DEVICE_URIS=cuda://0,cuda://1,cuda://0,cuda://1,cuda://0,cuda://1 \
scripts/run_qwen3_5_multimodal_cuda_workers.sh

The worker auto-prepare path also accepts repeated --image / --video from ember and ember_validate, so multi-image or mixed image+video prompts no longer need a manually prepared tensor bundle. For multistage multimodal artifacts, ember and ember_validate will also auto-route to the worker path when image/video inputs or a prepared --input-tensor-bundle are used, so --multistage-workers is now optional. Passing --device-uri cuda also keeps that worker auto-route in place, so it does not force multimodal multistage requests back onto the less stable single-process in-process path. The default build wrappers also stamp a concrete stage-to-GPU mapping into the artifact manifest, so --device-uris is optional unless you want to override placement.

For 4B, the proven path is multistage workers across two GPUs:

MANIFEST=/path/to/manifest.json \
VIDEO_PATH=/path/to/example.mp4 \
DEVICE_URIS=cuda://0,cuda://1,cuda://0,cuda://1,cuda://0,cuda://1,cuda://0,cuda://1 \
scripts/run_qwen3_5_4b_multimodal_cuda_workers.sh

Current validated coverage:

• Qwen3.5-0.8B: text, image, video on IREE
• Qwen3.5-0.8B: mixed image+video on IREE via the formal ember / ember_validate worker-routed entrypoints
• Qwen3.5-4B: image and video on IREE via multistage CUDA workers
• Qwen3.5-4B: mixed image+video on IREE via the formal ember / ember_validate worker-routed entrypoints

The remaining limitation is that single-process multi-GPU ember --device-uris ... is still less stable than the worker-based route.

Project layout

ember/
|-- apps/
|   |-- ember_cli/         # interactive/text generation entry
|   `-- ember_validate/    # structured validation entry
|-- backends/iree/         # IREE runtime/session/call glue
|-- cli/                   # shared CLI args
|-- core/                  # config, tokenizer, sampler, session
|-- runtime/               # runtime interface, driver, manifest contract
|-- scripts/               # export/build/run/verify helpers
|-- tests/                 # contract and smoke tests
`-- benchmarks/            # prefill/decode batch benchmarks

Runtime contract

Core types:

• runtime/driver.h
• runtime/artifact_manifest.h
• runtime/iruntime.h
• core/session.h

The current public execution ABI is:

• prefill
• decode
• reset_slots

The current shipped Qwen3 artifact still uses full-sequence-replay decode. That is correct for the current phase: the real next optimization step is to move from replay decode to true KV/stateful incremental decode without changing the outer runtime contract again.

Tests

ctest --test-dir build --output-on-failure

Current test ladder:

• manifest parsing
• driver parsing
• session slot lifecycle
• IREE load/unload smoke
• IREE prefill/decode smoke
• batched slot/reset smoke

Status

What is already true in this repo:

• IREE subproject build works
• ember, ember_validate, tests, and benchmarks compile
• local Qwen3 artifact path runs through the rewritten CLI

What is intentionally still phase-next work:

• true KV/stateful incremental decode
• varlen batch on a real Qwen3 artifact
• serving request queue / HTTP surface
• broader model export automation beyond the current Qwen3 path