GLiNER-Forge — Multi-Platform GLiNER Inference

Zero-shot NER (Named Entity Recognition) via GLiNER2, benchmarked across macOS (Apple Silicon), NVIDIA CUDA, and ARM edge (RK3588S).

Two Tracks

Quick Start

Python (MLX)

cd python
pip install -r requirements.txt
python gliner_mlx/infer.py

Python (ONNX)

python gliner_onnx/infer.py

Native build

cd native && ./build.sh
# → dist/libgliner_wrapper.dylib

Model

Using gliner2-multi-v1 (mDeBERTa-v3-base, 100+ languages).

Quantization Status

INT8 on RK3588S (NanoPi M6)

See reports/rk3588s_benchmark.md

⚠️ INT8 quantization pitfall: Running quantize_dynamic directly on fp16 ONNX will fail. You must first strip 153+ hidden fp16 tensors (Constant node attributes, ConstantOfShape, Cast targets). Use python/convert/fix_fp16.py to handle this.

Directory Layout

GLiNER-Forge/
├── python/
│   ├── gliner_mlx/      # MLX Python inference (OpenMed)
│   ├── gliner_onnx/     # ONNX Runtime inference
│   │   ├── infer.py             # macOS (CoreML)
│   │   └── infer_crossplatform.py # macOS + Linux ARM
│   ├── convert/         # to_onnx, to_mlx, quantize, fix_fp16
│   │   ├── fix_fp16.py          # fp16→fp32 cleanup (required before int8)
│   │   ├── quantize.py
│   │   ├── to_mlx.py
│   │   └── to_onnx.py
│   └── requirements.txt
├── native/
│   ├── src/             # gliner_wrapper.h/.cpp (MLX C++)
│   ├── dart/            # Dart FFI binding
│   ├── swift/           # Swift bridge
│   └── build.sh
├── bench/               # speed & accuracy benchmarks
│   ├── bench_rk3588s.py         # NanoPi M6 (RK3588S) benchmark
│   └── ...
├── reports/
│   └── rk3588s_benchmark.md     # RK3588S detailed report
├── models/              # weights (gitignored)
└── README.md

CUDA Results (Quadro T2000, 4GB VRAM)

See bench/results_cuda.md for full details.

Key finding: ONNX wins on Apple Silicon, PyTorch wins on CUDA.

Cross-Platform Comparison

INT8 Precision Note

RK3588S INT8 accuracy validated with Wikipedia Taiwan multilingual test data (18 cases, 10 languages: zh-TW, en, ja, ko, ru, fr, es, ar, th, vi). All cases achieve 100% recall at threshold 0.3. See bench/bench_rk3588s_accuracy.py.

Pipeline Recommendation by Platform

References

• GLiNER repo
• GLiNER2
• OpenMed MLX
• SemplificaAI ONNX
• gline-rs (Rust)
• dart-ml-forge — native wrapper pattern