Tencent Hy3 (HunYuan-3) — IQ4_NL GGUF

295B parameter Mixture-of-Experts model quantized to IQ4_NL (4.5 bits per weight), 158 GB.

⚠️ Requires a Custom Fork of ik_llama

This model will not load on standard llama.cpp or standard ik_llama.cpp. It requires a fork with hy_v3 architecture support:

→ https://github.com/noonr48/ik_llama-hy3 (branch: hy3-support)**

The fork adds:

• hy_v3 architecture enum, hparams, and tensor loading
• Graph builder dispatch (shares build_glm4_moe())
• Custom chat template handler with thinking token parsing
• MTP speculative decoding support (built-in NextN layer)
• EOS token leak fix

Standard llama.cpp does not recognize the hy_v3 architecture and will refuse to load this model.

Model Details

How This Model Was Made

1. Source model: Tencent HunYuan-3 (Apache 2.0 license), downloaded from HuggingFace in bf16 format
2. Conversion: Streaming HF → GGUF conversion using the custom fork's convert_hf_to_gguf.py (modified to detect and handle hy_v3 architecture)
3. Calibration: Importance matrix (imatrix) generated from a calibration dataset containing:
   • Open-source code samples (various languages, Apache/BSD/MIT licensed)
   • OEIS integer sequence data (public domain mathematical sequences)
   • General text passages
4. Quantization: IQ4_NL quantization applied using the imatrix for optimal weight selection (717 imatrix entries across 128 chunks)
5. Verification: Model loaded, inference tested, output quality verified

What's Tested and Working

Known Limitations

• EOS token ID override required: The GGUF metadata contains eos_token_id=3 (incorrect). You must pass --override-kv tokenizer.ggml.eos_token_id=int:120025 at runtime.
• 200K context is slow with CPU KV offload: With --no-kv-offload at 200K context, large prompts (3K+ tokens) process at ~5 tok/s due to PCIe bandwidth. Use 32K all-GPU KV for interactive use (~40 tok/s).
• MTP n_max=2 not recommended: Causes OOM on 24 GB GPUs with large prompts. n_max=1 is faster (higher acceptance rate) and more memory-efficient.
• --ubatch-size 512 can OOM with MTP: Use --ubatch-size 256 when MTP is enabled.

Quick Start

Build the fork

git clone -b hy3-support https://github.com/noonr48/ik_llama-hy3.git
cd ik_llama-hy3
cmake -B build -DGGML_CUDA=ON -DLLAMA_BUILD_TESTS=OFF
cmake --build build --target llama-server -j$(nproc)

Serve (32K context, recommended for interactive use)

./build/bin/llama-server \
  --model Hy3-IQ4_NL-v7.gguf \
  --host 0.0.0.0 --port 9999 \
  --n-gpu-layers 999 \
  -sm graph \
  --override-kv tokenizer.ggml.eos_token_id=int:120025 \
  --ctx-size 32768 --batch-size 512 --ubatch-size 256 \
  --flash-attn on --cache-type-k f16 --cache-type-v f16 \
  --jinja --chat-template-file models/templates/Hy3.jinja \
  --reasoning-format deepseek --reasoning on \
  --spec-type mtp:n_max=1,p_min=0.0

Serve (200K context, for batch / long-context tasks)

./build/bin/llama-server \
  --model Hy3-IQ4_NL-v7.gguf \
  --host 0.0.0.0 --port 9999 \
  --n-gpu-layers 999 \
  --no-kv-offload \
  --override-kv tokenizer.ggml.eos_token_id=int:120025 \
  --ctx-size 200000 --batch-size 512 --ubatch-size 256 \
  --flash-attn on --cache-type-k f16 --cache-type-v f16 \
  --jinja --chat-template-file models/templates/Hy3.jinja \
  --reasoning-format deepseek --reasoning on \
  --spec-type mtp:n_max=1,p_min=0.0

MTP Speculative Decoding

Hy3 includes a built-in Multi-Token Prediction layer (block 80, NextN architecture). When enabled, the model predicts the next token using a lightweight draft pass, then verifies it with the full model. This reduces the number of full forward passes by ~30-50%.

Enable with: --spec-type mtp:n_max=1,p_min=0.0

Measured speedup: ~54% faster on large prompts (3K+ tokens), ~33% faster on short prompts.

Hardware Requirements

• Minimum: ~160 GB total VRAM (e.g., 7× RTX 3090, or equivalent)
• Tested on: 10-GPU mixed fleet (1× RTX 5090, 3× RTX 3090, 6× RTX 5060 Ti), ~199 GB total VRAM
• CPU RAM: 64 GB minimum for model loading (more if using CPU KV offload)

Chat Template

A custom Jinja2 chat template is required for correct thinking token handling. The template handles:

• <think:opensource></think:opensource> thinking markers
• <｜hy_eos:opensource｜> EOS token
• System/user/assistant message formatting

The template file (models/templates/Hy3.jinja) is included in the GitHub fork repository.

Credits

• Original model: Tencent HunYuan-3 (Apache 2.0)
• Inference engine: ik_llama.cpp by Ikawrakow
• hy_v3 fork: noonr48/ik_llama-hy3
• MTP infrastructure: Originally built for GLM-DSA / Qwen3.5-MoE in ik_llama.cpp

License

Apache 2.0 (inherited from the original Tencent HunYuan-3 model)