Inkling-MXFP4

Model Overview

• Model Architecture: Thinking Machines Lab Inkling
  • Input: Text, Image, Audio
  • Output: Text
• Inference Engine: TokenSpeed
• Model Optimizer: AMD Quark (0.12.post1+rocm72.torch2.11)
  • Quantized layers: MoE routed experts only
  • Weight quantization: OCP MXFP4, static
  • Activation quantization: OCP MXFP4, dynamic

This model was built by applying AMD Quark MXFP4 quantization to the BF16 Thinking Machines Lab Inkling checkpoint. The quantization targets the MoE routed experts, while attention layers and shared experts are kept in BF16.

Environment

The quantization workflow was prepared on an AMD gfx950 system. The inspected container environment was:

• GPU: AMD MI350/MI355
• Target graphics version: gfx950
• ROCm: 7.2.1
• amdgpu driver: 6.16.13
• OS: Linux 6.8.0-84, x86_64
• Python: 3.12.3
• PyTorch: 2.13.0+rocm7.1
• AMD Quark: 0.12.post1+rocm72.torch2.11
• Safetensors: 0.8.0
• Transformers: 5.13.1

Create and activate the Quark environment:

python3 -m venv ~/.venv-quark
source ~/.venv-quark/bin/activate

Install the required packages:

python -m pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm7.1
python -m pip install amd-quark --extra-index-url https://pypi.amd.com/quark/rocm72/simple
python -m pip install safetensors transformers accelerate tqdm

Model Quantization

The model was quantized with the Quark file-to-file flow. This avoids loading the full BF16 checkpoint into GPU memory at once, which is important for very large MoE checkpoints. Run the quantization script:

python quantize_quark.py \
  --model_dir /path/to/model \
  --output_dir /path/to/output \
  --quant_scheme mxfp4 \
  --file2file_quantization

The script applies the model-specific exclusion policy automatically in file-to-file mode. The resulting checkpoint stores MXFP4 routed-expert weights and scales while preserving non-routed-expert components in BF16.

Deployment

This model can be served with TokenSpeed:

tokenspeed serve \
  --model lightseekorg/Inkling-MXFP4 \
  --attn-tp-size 4 \
  --moe-tp-size 4 \
  --max-model-len 81920 \
  --max-num-seqs 16 \
  --max-prefill-tokens 8192 \
  --chunked-prefill-size 8192 \
  --gpu-memory-utilization 0.95 \
  --disable-cuda-graph-padding \
  --trust-remote-code \
  --dtype bfloat16 \
  --disable-kvstore \
  --kvstore-ratio 0 \
  --block-size 128 \
  --host 127.0.0.1 \
  --port 22015

Evaluation

The following validation results are placeholders and will be updated before public release.