📄 简体中文 | ✨ New Project: AI-Enhancement-Filter (powered by onnx-tool)

---

onnx-tool

A comprehensive toolkit for analyzing, optimizing, and transforming ONNX models with advanced capabilities for LLMs, diffusion models, and computer vision architectures.

• LLM Optimization: Build and profile large language models with KV cache analysis (example)
• Graph Transformation:
  • Constant folding (docs)
  • Operator fusion (docs)
• Advanced Profiling:
  • Rapid shape inference
  • MACs/parameter statistics with sparsity awareness
• Compute Graph Engine: Runtime shape computation with minimal overhead (details)
• Inference Engine: PyTorch-backed graph inference with memory pool (details)
  • 40+ registered op kernels (Conv, Add, Relu, Gemm, etc.)
  • Up to 11.3x faster than PyTorch in dynamic resolution scenarios
  • 54% memory reduction via two-pass compression algorithm
• Memory Compression:
  • Activation memory optimization (up to 95% reduction)
  • Weight quantization (FP16, INT8/INT4 with per-tensor/channel/block schemes)
• Quantization & Sparsity: Full support for quantized and sparse model analysis

🤖 Supported Model Architectures

🆕 Qwen3.5 Series: Full support for Qwen3.5 hybrid architecture including:

• Gated DeltaNet (GDN) layers with linear attention
• QKV Gating (Q projection with built-in gate, applied before O-projection)
• Sparse Mixture-of-Experts (MoE) with routed + shared experts
• Mixed layer types (linear_attention / full_attention) per config
• Multimodal Vision Encoder (ViT + MLP Projector) with resolution-aware profiling

---

⚡ Build & Profile LLMs in Seconds

Profile 10 Hugging Face models in under one second. Export ONNX models with llama.cpp-like simplicity (code).

Model Statistics (1k token input)

Multimodal: Vision Encoder vs LLM MACs 🆕

Vision encoder (ViT-24L + MLP Projector) vs LLM backbone. LLM input seq_len = vision patches + 1K text tokens. Both models share the same ViT architecture. See benchmark/vision_llm_compare.py.

💡 Vision MACs scale with O(p²), LLM MACs with O(t²) where t = patches + text. At 448×448, vision is ~4-5% of LLM; at 1344×1344, ~12-13%. Vision is always the minor component.

MoE Activated Parameters vs Sequence Length

Activated parameters = sum of all nodes' static_params (weights actually accessed during forward pass). S=32: Qwen3.5-35B-A3B & MiniMax-M2.7 experts fully activated (gap ≈ embedding); DeepSeek-V4 not yet saturated (top-6 × 32 = 192 < num_experts).

Latency Estimation (4-bit weights, 16-bit KV cache)

Prefill Throughput (tokens/s, 1k input)

Decode Throughput (tokens/s)

💡 Latencies computed from hardware specs – no actual inference required. Uses BF16/FP16 compute with FP32 accumulate as the standard.

---

🔧 Basic Parsing & Editing

Intuitive API for model manipulation:

from onnx_tool import Model

model = Model('model.onnx')          # Load any ONNX file
graph = model.graph                  # Access computation graph
node = graph.nodemap['Conv_0']       # Modify operator attributes
tensor = graph.tensormap['weight']   # Edit tensor data/types
model.save_model('modified.onnx')    # Persist changes

See comprehensive examples in benchmark/examples.py.

---

📊 Shape Inference & Profiling

All profiling relies on precise shape inference:

Profiling Capabilities

• Standard profiling: MACs, parameters, memory footprint
• Sparse-aware profiling: Quantify sparsity impact on compute

📚 Learn more:

• Profiling Guide
• PyTorch Integration
• TensorFlow Integration

---

⚙️ Compute Graph & Shape Engine

Transform exported ONNX graphs into efficient Compute Graphs by removing shape-calculation overhead:

• Compute Graph: Minimal graph containing only compute operations
• Shape Engine: Runtime shape resolver for dynamic models

Use Cases:

• Integration with custom inference engines (guide)
• Shape regression testing (example)

---

💾 Memory Compression

Activation Memory Compression

Reuses temporary buffers to minimize peak memory usage – critical for LLMs and high-res CV models.

✅ Typical models achieve >90% activation memory reduction
📌 Implementation: benchmark/compression.py

The compress_memory() algorithm has been patched with two improvements (see onnx_tool/graph.py):

1. Size-sorted allocation: New tensors within each node are allocated in descending size order, reducing fragmentation
2. Tail compression: Unused space at the end of the memory pool is trimmed
3. List reference fix: Each tensor's [offset, size] is stored as an independent copy, preventing accidental cross-tensor aliasing

Benchmark results across models:

✅ Optimized algorithm achieves up to 54% additional pool reduction (ResNet50: 21.4→10.7 MB vs original)

Inference Engine

The inference/ module provides a complete PyTorch-backed inference engine built on the compressed memory pool:

• MemoryPool: Zero-copy tensor views into a pre-allocated contiguous buffer
• Kernel Registry: 40+ registered op kernels (Conv, Add, Relu, Gemm, MatMul, etc.)
• GraphInfer: Graph-level inference with shape engine integration

Performance highlights (ResNet18 on Intel XPU):

📌 See inference/README.md for full benchmark details

Weight Compression

Essential for deploying large models on memory-constrained devices:

Supported schemes:

• ✅ FP16
• ✅ INT8: symmetric/asymmetric × per-tensor/channel/block
• ✅ INT4: symmetric/asymmetric × per-tensor/channel/block

📌 See benchmark/examples.py for implementation examples.

---

🚀 Installation

# PyPI (recommended)
pip install onnx-tool

# Latest development version
pip install --upgrade git+https://github.com/ThanatosShinji/onnx-tool.git

Requirements: Python ≥ 3.6

⚠️ Troubleshooting: If ONNX installation fails, try:

pip install onnx==1.8.1 && pip install onnx-tool

---

Known Issues

• Loop op is not supported
• Sequence type is not supported

---

📈 Model Zoo Results

Comprehensive profiling of ONNX Model Zoo and SOTA models. Input shapes defined in data/public/config.py.

📥 Download pre-profiled models (with full tensor shapes):

• Baidu Drive (code: p91k)
• Google Drive

🤝 Contributing

Contributions are welcome! Please open an issue or PR for:

• Bug reports
• Feature requests
• Documentation improvements
• New model support