---

📰 Latest News

---

📖 Introduction

LLM-TPU is an open-source project officially maintained by SOPHGO, dedicated to deploying mainstream generative AI models (LLM / VLM) on SOPHGO BM1684X / BM1688 / CV186X series TPU chips.

• 🚀 One-click compilation: llm_convert.py exports HuggingFace weights directly to bmodel
• 🧩 Rich model coverage: Dozens of models including Qwen / Llama / DeepSeek / InternVL / MiniCPM / Phi / ChatGLM
• 🎯 Multimodal: Inference across text, image, video, and audio
• ⚡ Efficient inference: AWQ/GPTQ quantized models, dynamic compilation, KV Cache, multi-chip parallelism
• 🛠️ Dual-language demos: Popular models ship with both Python and C++ reference implementations
• 📦 Ready to use: Pre-compiled bmodels available for download — no compilation required

Compiling models requires the TPU-MLIR environment (Docker or source build both work). Alternatively, use the pre-compiled bmodels provided in each demo. See models/ for the full model list.

---

🚀 Quick Start

Get an LLM running on your TPU device in just two steps:

git clone https://github.com/sophgo/LLM-TPU.git
cd LLM-TPU
./run.sh --model qwen3.5

One-command demo models

---

🧠 Supported Models

Multimodal Models (VLM / Audio / Vision)

LLM Models

Full Directory Index

The models/ directory currently contains the following model implementations:

LLM: ChatGLM3 · Llama3 · MiniCPM4 · Phi-3 · Qwen2_5 · Qwen3 · Qwen3_Embedding

Multimodal (Vision / Video / Audio): Falcon-Perception · Gemma3 · Gemma4 · GLM4V · InternVL3 · Janus-Pro · Llama3_2-Vision · LocateAnything · Mage_VL · MiniCPMV4 · MiniCPMV4_6 · NVILA · Qwen2_5_Omni · Qwen2_5_VL · Qwen2_VL · Qwen3_5 · Qwen3_ASR · Qwen3_TTS · Qwen3_VL · Step3_VL

Legacy

Older demos built with the pre-llm_convert.py compile flow (ONNX export + model_transform.py / model_deploy.py) are kept under models/legacy/ for reference and are no longer actively maintained:

LLM: Baichuan2 · ChatGLM2 · CodeFuse · DeepSeek-V2 · GLM4 · Llama2 · LWM · Megrez · MiniCPM3 · Mistral · Qwen · Qwen1_5 · Qwen2 · RWKV6 · RWKV7 · WizardCoder · Yi · Yi34B

Multimodal: DriveMM · InternVL2 · MiniCPM-V-2_6 · Molmo · Qwen2_Audio · VILA1_5

See each subdirectory for complete source code and conversion details.

---

🧩 LLM Compilation Flow

Using Qwen3.5-2B as an example:

1. Download Weights

Prefer AWQ / GPTQ / AutoRound quantized versions for better accuracy.

git lfs install
git clone https://huggingface.co/Intel/Qwen3.5-2B-int4-AutoRound

2. Set Up TPU-MLIR

Refer to TPU-MLIR

3. One-click Compilation to bmodel

llm_convert.py \
    -m /workspace/Qwen3.5-2B-int4-AutoRound \
    -s 2048 --max_input_length 1024 \
    -c bm1684x \
    -o qwen3.5_2b

Two compile modes: without history vs. with history

All LLM compile scenarios fall into two categories, controlled by --use_history_kv:

Mode 1 — Without history. Typical command:

llm_convert.py -m Qwen3.5-2B-int4-AutoRound -c bm1688 -s 2048 --max_input_length 1024 --out_dir qwen3_5_bm1688

• Compiles two kinds of nets: block_* (prefill) and block_cache_* (decode).
• -s sets the maximum total length; --max_input_length sets the maximum single-input length.
• Recommended for single-turn conversations with short contexts (e.g. within 4K).

Mode 2 — With history. Typical command:

llm_convert.py -m Qwen3.5-2B-int4-AutoRound -c bm1688 -s 8192 --use_history_kv --chunk_length 1024 --out_dir qwen3_5_bm1688

• Compiles three kinds of nets: block_* (prefill), block_kv_* (prefill with history KV), and block_cache_* (decode).
• -s sets the maximum total length; --chunk_length sets the segment length used for chunked inference. For example, with --chunk_length 1024 and a 7K-token input, prefill runs in 7 chunk passes: the first through block_, the remaining 6 through block_kv_. Decode is also segmented by KV-cache length, so performance at 1K / 2K / 4K / 8K varies with the context length.
• Recommended whenever multi-turn history is needed, contexts are long (e.g. 8K), or you are unsure — it is more flexible while retaining good performance.

Key llm_convert.py Arguments

💡 Choosing quantization: if the model is already quantized, you do NOT need to specify quantize; unquantized models require it.

For advanced options (--quantize, --q_group_size, --max_pixels, --embedding_disk, --lora_max_rank), see Advanced Compile Options; for more capabilities, see Advanced Features.

Once finished, the output directory will contain the corresponding bmodel and config directory, ready to load for inference.

---

⚙️ Advanced Features

Advanced Compile Options

Less commonly used llm_convert.py arguments:

---

Using the Demo

The interactive demos support a few convenience inputs:

• Slash commands — enter /exit (or /q, /quit) to quit the demo, and /clear (or /new) to start a new chat session.
• @<path> attachments — include @<path> in your question to attach a file:
  • Images (and videos, where the model supports them), e.g. what is the image about? @./test.jpg
  • Text files (.txt / .md), e.g. what is it talking about? @./story.txt

---

🎯 Accuracy Optimization Tips

1. Prefer AWQ / GPTQ / AutoRound quantized models when converting to bmodel — they incur the least accuracy loss.
2. If only floating-point weights are available, first apply W4A16 quantization with AutoAWQ or AutoGPTQ, then compile to bmodel.

---

❓ FAQ

Please refer to the LLM-TPU FAQ.

---

🔗 Resources

• 📄 An MLIR-Based Compilation Method for Large Language Models — Paper describing the TPU-MLIR LLM compilation flow
• 📘 TPU-MLIR — Main compiler repository
• 📗 TPU-MLIR Quick Start Guide
• 🎬 TPU-MLIR Paper / Full Project Walkthrough (Bilibili)
• ✍️ ChatGLM2 Pipeline Analysis & TPU-MLIR Deployment (Zhihu)
• 🌐 SOPHGO Official Website

---

🤝 Contributing & Feedback

Issues and suggestions are welcome via GitHub Issues, and Pull Requests are appreciated to help grow the ecosystem. If you are interested in SOPHGO chips or business cooperation, feel free to reach out through the SOPHGO website.

📄 License

This project is open-sourced under the Apache 2.0 license. See third-party-licenses/ for third-party component licenses.