GLiNER Guard Serving Experiment

Benchmarking GLiNER2 guard models for content moderation serving:

• Ray Serve vs LitServe throughput/latency comparison
• Label scaling: accuracy and latency with 6 → 56 labels
• Bi-encoder support with label caching
• Dynamic batching (B1-B8 sweep)
• REST vs gRPC protocol comparison

Quick Start

# Clone
git clone https://github.com/Stepan5024/gliner-guard-experiment.git
cd gliner-guard-experiment

# Install GLiNER2
cd GLiNER2
pip install -e .
pip install scipy

# Run accuracy benchmark (GPU recommended)
python benchmarks/benchmark_accuracy_vs_labels.py

# Run latency benchmark
python benchmarks/benchmark_label_scaling.py

# Run tests
pytest tests/ -k "not slow" -v

Structure

GLiNER2/                    # GLiNER2 fork with bi-encoder support
├── gliner2/
│   ├── model.py            # Extractor with bi-encoder layers
│   ├── processor.py        # Sequence length guard
│   ├── inference/
│   │   ├── engine.py       # GLiNER2 main class
│   │   └── schema_registry.py  # Dynamic label registry
│   └── eval/
│       └── evaluate.py     # F1/P/R evaluation harness
├── benchmarks/
│   ├── benchmark_accuracy_vs_labels.py
│   ├── benchmark_label_scaling.py
│   └── benchmark_batching.py
└── tests/

gliner-guard-serve/         # Ray Serve deployment + load testing
├── ray-serve/              # Ray Serve app (REST + gRPC)
├── litserve-baseline/      # LitServe baseline
├── test-script/            # Locust load tests + datasets
├── scripts/                # Automation scripts
├── results/                # Benchmark outputs
├── docker-compose.yml      # 3 profiles: litserve, ray-serve, ray-serve-grpc
└── Makefile                # 22 automation targets

GPU Requirements

• Smoke tests: RTX 4090 (24GB) — ~$0.59/hr
• Final benchmarks: A100 80GB — ~$1.80/hr
• Total benchmark time: ~39 hours (automated)