中文

RKNN3-MODEL-ZOO

Table of Contents

• 1. Introduction
• 2. Supported Models
  • 2.1 LLM
  • 2.2 VLM
  • 2.3 Omni
  • 2.4 ASR (Speech Recognition)
  • 2.5 TTS (Text-to-Speech)
  • 2.6 Embedding / Reranker
  • 2.7 Translation
  • 2.8 OCR
  • 2.9 CV (Computer Vision)
  • 2.10 Pre-converted RKNN Models
• 3. Supported Platforms
• 4. Quick Start
  • 4.1 Environment Setup
  • 4.2 General Deployment Flow
  • 4.3 Common Parameters
• 5. Advanced Features
  • 5.1 SpeedUP Inference Acceleration
  • 5.2 Multi-card Inference
  • 5.3 LoRA Support
• 6. Model Adaptation Guide
• 7. Important Notes
• 8. Additional Notes

1. Introduction

RKNN3 SDK provides the complete software stack for deploying AI models on RK1820/RK1828 coprocessors, including:

• RKNN3-Toolkit: PC-side software development kit for model conversion, inference, performance evaluation, etc.
• RKNN3 Runtime: On-board runtime library providing C/C++ programming interfaces for deploying RKNN models and accelerating AI applications.
• RKNN3 Model Zoo: Model conversion and deployment example repository, including reference implementations for CNN / LLM / VLM and other models.

This repository provides a complete model deployment workflow:

• Model Export: Export HuggingFace / PyTorch models to ONNX format
• Model Conversion: Convert ONNX models to RKNN format using RKNN3 Toolkit
• On-board Deployment: Provide C++ inference example code

2. Supported Models

2.1 LLM

2.2 VLM

2.3 Omni

2.4 ASR (Speech Recognition)

2.5 TTS (Text-to-Speech)

2.6 Embedding / Reranker

2.7 Translation

2.8 OCR

2.9 CV (Computer Vision)

2.10 Pre-converted RKNN Models

Users can download pre-converted RKNN models from the RKNN3_SDK cloud drive (access code: rknn). The models for this release are available under RKNN3_SDK/rknn3_models/v1.1.0.

3. Supported Platforms

Build and runtime library notes:

• The top-level build scripts build-linux.sh / build-android.sh accept -t with rk3588, rk3576, rk3572, and x86.
• RKNN3 runtime libraries installed into each demo's lib/ directory are SoC-specific:
  • RK3588 / RK3576: librknn3_api.so and librknn3_api_rkcp.so
  • RK3572: librknn3_api.so and librknn3_api_native.so

4. Quick Start

4.1 Environment Setup

Requirements: Python 3.10

cd rknn3_model_zoo/
pip install -r requirements.txt
export PYTHONPATH=./

4.2 General Deployment Flow

All models follow a unified four-step flow: Export ONNX -> Convert RKNN -> Build -> Run.

LLM Model (Example: Qwen2.5-3B)

# Step 1: Export ONNX model (--quant enables GRQ quantization)
cd examples/Qwen2_5/python/
python export_llm.py --quant

# Step 2: Convert to RKNN model
python export_rknn.py

# Step 3: Build
cd ../../../
export GCC_COMPILER=<GCC_COMPILER_PATH>     # Optional: specify cross-compiler path
./build-linux.sh -t rk3588 -a aarch64 -d Qwen2_5

# Step 4: Push and run
adb push install/rk3588_linux_aarch64/rknn_Qwen2_5_demo/ /data/
adb shell
cd /data/rknn_Qwen2_5_demo
export LD_LIBRARY_PATH=./lib
./rknn_qwen2_5_demo \
    model/Qwen2.5-0.5B-Instruct.rknn \
    model/Qwen2.5-0.5B-Instruct.weight \
    model/Qwen2.5-0.5B-Instruct.tokenizer.gguf \
    model/Qwen2.5-0.5B-Instruct.embed.bin \
    0xff \
    "Who are you?"

VLM Model (Example: Qwen3-VL-4B)

# Step 1: Export ONNX models (Vision + LLM exported separately; --quant requires calibration data generated first)
cd examples/Qwen3_VL/python/llm/
python make_calidata.py --model_path Qwen/Qwen3-VL-4B-Instruct --modelscope
python export_llm.py --quant \
    --model_path Qwen/Qwen3-VL-4B-Instruct \
    --export_llm_path Qwen3-VL-4B-llm.onnx --modelscope
cd ../vision/
python make_calidata.py --model_path Qwen/Qwen3-VL-4B-Instruct --modelscope
python export_vision.py --quant \
    --model_path Qwen/Qwen3-VL-4B-Instruct \
    --export_vision_path Qwen3-VL-4B-vision.onnx --modelscope

# Step 2: Convert to RKNN models (Vision + LLM converted separately)
python export_rknn.py \
    --onnx_path Qwen3-VL-4B-vision.onnx \
    --rknn_path Qwen3-VL-4B-vision.rknn --platform rk1820   # Vision RKNN
cd ../llm/
python export_rknn.py \
    --onnx_path Qwen3-VL-4B-llm.onnx \
    --config Qwen3-VL-4B-llm.config.pkl \
    --rknn_path Qwen3-VL-4B-llm.rknn --platform rk1820       # LLM RKNN

# Step 3: Build
cd ../../../
export GCC_COMPILER=<GCC_COMPILER_PATH>     # Optional
./build-linux.sh -t rk3588 -a aarch64 -d Qwen3_VL

# Step 4: Push and run
adb push install/rk3588_linux_aarch64/rknn_Qwen3_VL_demo/ /data/
adb shell
cd /data/rknn_Qwen3_VL_demo
export LD_LIBRARY_PATH=./lib
./rknn_qwen3_vl_demo \
    model/Qwen3-VL-4B-vision.rknn model/Qwen3-VL-4B-vision.weight \
    model/Qwen3-VL-4B-llm.rknn model/Qwen3-VL-4B-llm.weight \
    model/Qwen3-VL-4B-llm.tokenizer.gguf model/Qwen3-VL-4B-llm.embed.bin \
    0xff 0xff \
    model/demo.jpg \
    "Please describe this image"

For model-specific parameters and differences, refer to the README in each example directory.

4.3 Common Parameters

The export scripts of each model share the following common parameters (specific default values vary by model; see the README in each example directory):

Notes:

• When using GRQ quantization, the model contains quantization parameters; no quantization dataset is needed for RKNN conversion
• RKNN conversion uses weight-separated mode, generating both .rknn and .weight files
• LLM model export also includes Config (.config.pkl), Tokenizer (.tokenizer.gguf), and Embed (.embed.bin) files

5. Advanced Features

5.1 SpeedUP Inference Acceleration

The Qwen2.5-VL and Qwen3-VL examples can link against the SpeedUP third-party library for inference acceleration.

File Location

Keep the following files in the release package:

3rdparty/SpeedUP/
├── include/speedup.h
├── Linux/aarch64/libSpeedUP.so
└── Android/arm64-v8a/libSpeedUP.so

Build

# Qwen2.5-VL
./build-linux.sh -t rk3588 -a aarch64 -d Qwen2_5_VL

# Qwen3-VL
./build-linux.sh -t rk3588 -a aarch64 -d Qwen3_VL

After installation, libSpeedUP.so is copied into the corresponding demo lib/ directory.

Runtime Arguments

./rknn_qwen2_5_vl_demo \    # or ./rknn_qwen3_vl_demo
    <vision_model_path> <vision_weight_path> \
    <llm_model_path> <llm_weight_path> \
    <tokenizer_path> <embedding_path> \
    <vision_core_mask> <llm_core_mask> \
    <image_path> <prompt> \
    [model_width model_height] [speedup_ratio]

speedup_ratio is optional:

For RKNN3 multi-core devices, this is usually suitable:

0xff 0xff

Examples

Qwen2.5-VL:

cd /userdata/rknn3-model-zoo/install/rk3588_linux_aarch64/rknn_Qwen2_5_VL_demo
export LD_LIBRARY_PATH=./lib:$LD_LIBRARY_PATH

./rknn_qwen2_5_vl_demo \
    /userdata/Qwen2.5-VL-3B/Qwen2.5-VL-3B-vision.rknn \
    /userdata/Qwen2.5-VL-3B/Qwen2.5-VL-3B-vision.weight \
    /userdata/Qwen2.5-VL-3B/Qwen2.5-VL-3B-llm.rknn \
    /userdata/Qwen2.5-VL-3B/Qwen2.5-VL-3B-llm.weight \
    /userdata/Qwen2.5-VL-3B/Qwen2.5-VL-3B-llm.tokenizer.gguf \
    /userdata/Qwen2.5-VL-3B/Qwen2.5-VL-3B-llm.embed.bin \
    0xff 0xff \
    /userdata/rknn3-model-zoo/examples/Qwen2_5_VL/data/vision/demo.jpg \
    "Describe this image" \
    392 392 \
    1.0

Qwen3-VL:

cd /userdata/rknn3-model-zoo/install/rk3588_linux_aarch64/rknn_Qwen3_VL_demo
export LD_LIBRARY_PATH=./lib:$LD_LIBRARY_PATH

./rknn_qwen3_vl_demo \
    /userdata/Qwen3-VL-model/Qwen3-VL-4B-vision_384_384.rknn \
    /userdata/Qwen3-VL-model/Qwen3-VL-4B-vision_384_384.weight \
    /userdata/Qwen3-VL-model/Qwen3-VL-4B-llm.rknn \
    /userdata/Qwen3-VL-model/Qwen3-VL-4B-llm.weight \
    /userdata/Qwen3-VL-model/Qwen3-VL-4B-llm.tokenizer.gguf \
    /userdata/Qwen3-VL-model/Qwen3-VL-4B-llm.embed.bin \
    0xff 0xff \
    /userdata/rknn3-model-zoo/examples/Qwen2_5_VL/data/vision/demo.jpg \
    "Describe this image" \
    384 384 \
    1.0

5.2 Multi-card Inference

This example splits an LLM model into multiple segments (stages) at Transformer layer boundaries, with each segment deployed on one RK182X accelerator card. Through pipeline parallelism, it enables multi-card collaborative inference, supporting larger models and improving prefill performance.

Currently supported models:

Model Splitting Principle

Segments are cut between layers without changing the intra-layer computation order; the last segment always contains the final norm and lm_head on top of its assigned Transformer layers. The export script does not simply split by total_layers / num_segments — it estimates each layer's weight (Transformer layers as W4A16/group32, lm_head as W6A16/group32, final norm as FP16) and automatically searches for balanced split boundaries so the estimated total weight of every segment is as close as possible. --num_segments N specifies the number of segments.

Model Conversion

# Example: splitting Qwen3.5-9B into 2 segments
cd examples/multicard/python/qwen3_5

# Export segmented ONNX model
python export_llm_segment.py --model_path /path/to/Qwen3.5-9B --multi_segment --num_segments 2

# Export segmented RKNN model
python export_rknn_segment.py --multi_segment --num_segments 2

Qwen3.5 and Gemma-4 use an external rope cache by default. Each segment produces an identical .safetensors file; pass any one of them via --rope-tensor at runtime.

Build and Run

# Build
./build-linux.sh -t rk3588 -a aarch64 -d multicard

# Run (2-segment Qwen3.5-9B)
cd /data/rknn_multicard_demo
export LD_LIBRARY_PATH=./lib
taskset f0 ./rknn_multicard_demo \
    --model /data/models/multicard/Qwen3.5-9B-llm_seg0.rknn \
    --weight /data/models/multicard/Qwen3.5-9B-llm_seg0.weight \
    --vocab /data/models/multicard/Qwen3.5-9B-llm.tokenizer.gguf \
    --embed /data/models/multicard/Qwen3.5-9B-llm.embed.bin \
    -c 4096 \
    --core-mask 0xff \
    --stage-count 2 \
    --bucket-size 128 \
    --rope-tensor /data/models/multicard/Qwen3.5-9B-llm_seg0.safetensors \
    --prompt "<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\n" \
    --predict 128

Only the seg0 model/weight paths need to be passed; the paths of seg1..segN are derived automatically from the _segN suffix.

For complete parameter descriptions, Gemma-4 examples, KV Cache rebuild, and multi-card inference code logic, see examples/multicard/README.md.

5.3 LoRA Support

RKNN3 supports LoRA (Low-Rank Adaptation) adapter loading, which overlays LoRA weights onto the RKNN model without modifying the base model weights, enabling task-specific fine-tuned inference. Currently, Qwen3-VL is provided as an example.

Model Export

LoRA weights (e.g., adapter_model.safetensors) do not need to be converted to ONNX separately; they can be passed directly to export_rknn.py:

cd examples/Qwen3_VL_LoRA/python/llm

# Export RKNN model with LoRA
python export_rknn.py \
    --onnx_path Qwen3-VL-4B-llm.onnx \
    --config Qwen3-VL-4B-llm.config.pkl \
    --rknn_path Qwen3-VL-4B-llm-lora.rknn \
    --lora_path /path/to/lora/adapter_model.safetensors \
    --lora_config_path /path/to/lora/adapter_config.json

After export, two files are generated: .rknn and .lora_weight. Both must be provided during C++ inference.

Build and Run

# Build
./build-linux.sh -t rk3588 -a aarch64 -d Qwen3_VL_LoRA

# Run (Base + LoRA dual-path comparison)
cd /data/rknn_Qwen3_VL_LoRA_demo
export LD_LIBRARY_PATH=./lib
./rknn_qwen3_vl_demo \
    ./model/vision.rknn ./model/vision.weight \
    ./model/llm.rknn ./model/llm.weight \
    ./model/tokenizer.gguf ./model/embed.bin \
    0x3 0x3 \
    ./model/demo.jpg "Describe this image" \
    384 384 \
    2048 3072 \
    ./model/llm_lora.weight

The program will sequentially output inference results and performance statistics for both the Base model and LoRA model.

LoRA-related API

Recommended call sequence: lora_init -> query(LORA_NUM) -> query(LORA_INFO) -> lora_load -> session_enable_lora -> lora_unload

For complete parameter descriptions, API details, and dual-path inference implementation, see examples/Qwen3_VL_LoRA/README.md.

6. Model Adaptation Guide

• Same-series Compatibility: Examples within the same model series are interchangeable. For instance, the Qwen2.5-0.5B example works directly with Qwen2.5-7B by simply changing the model loading path.

• New Model Adaptation: For LLM models not included in this repository, refer to the LLM Model Adaptation Tutorial for ONNX export and deployment porting.

7. Important Notes

• Transformers Version: Different models may require different transformers versions. Before exporting to ONNX, install the correct version. Version info can be found in the transformers_version field of the model's config.json (e.g., https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/blob/main/config.json ). Some models have special version requirements; refer to the requirements.txt in each example directory.

• PyTorch Version: Recommended PyTorch <= 2.8.0 (Qwen3-VL, Gemma-4, and other models require PyTorch >= 2.9.0; PaddleOCR-VL requires transformers == 4.55.0. See the requirements.txt under the corresponding model for details.)

• Module Compatibility: Gemma-4 Audio and LLM models must use the same version (both E2B or both E4B); mixing versions is not supported.

8. Additional Notes

This repository uses the following mirror sites by default to obtain model files:

• ModelScope (recommended for China users)
• HuggingFace Mirror