GLiNER2 API (Jetson)

A FastAPI server for GLiNER2 multi-task information extraction, built specifically for NVIDIA Jetson.

Built on JetPack 6 / L4T R36, and verified running on JetPack 7 / L4T R39 (the image carries its own CUDA 12.6 userspace, so no rebuild is needed — see JETSON.md).

Supports entity extraction, text classification, structured JSON extraction, and combined multi-task schemas — all running on-device with GPU acceleration.

Quick Start (Local)

make install
make run-no-preload

API will be at: http://localhost:8000

For development with auto-reload (CPU-only, restarts on file changes):

make dev

Quick Start (Docker)

make docker-build-no-cache  # full rebuild (first time or after Dockerfile changes)
make docker-build            # cached rebuild
make docker-run              # foreground with GPU

Endpoints

Example

curl -X POST http://localhost:8000/extract_entities \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Patient received 400mg ibuprofen for severe headache at 2 PM.",
    "labels": ["medication", "dosage", "symptom", "time"]
  }'

Response:

{
  "entities": {
    "medication": ["ibuprofen"],
    "dosage": ["400mg"],
    "symptom": ["headache"],
    "time": ["2 PM"]
  }
}

Multi-task

/extract_multitask takes a single schema_config object — not top-level entities / classification keys:

curl -X POST http://localhost:8012/extract_multitask \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Apple CEO Tim Cook announced record revenue in Cupertino.",
    "schema_config": {
      "entities": ["company", "person", "location"],
      "classification": {"name": "sentiment", "labels": ["positive", "negative"]},
      "structure": {
        "name": "announcement",
        "fields": [{"name": "who", "dtype": "str"}, {"name": "what", "dtype": "str"}]
      }
    }
  }'

{
  "entities": {"company": ["Apple"], "person": ["Tim Cook"], "location": ["Cupertino"]},
  "sentiment": "positive",
  "announcement": [{"who": "Tim Cook", "what": "record revenue"}]
}

Environment Variables

Concurrency

Inference endpoints use asyncio.to_thread() to run model inference off the main event loop. This keeps the server responsive to health checks and new connections while a request is being processed.

To keep behavior stable under load, the API now includes:

• lock-protected model initialization (prevents duplicate first-load races)
• bounded inference concurrency with a semaphore
• request timeouts for long-running inference calls
• payload size/shape limits for text, labels, and schema fields

Status behavior under load or oversized payloads:

• 413 when text exceeds MAX_TEXT_CHARS
• 503 when no inference slot is available within INFERENCE_ACQUIRE_TIMEOUT_SECONDS
• 504 when inference exceeds REQUEST_TIMEOUT_SECONDS

For higher throughput, tune MAX_CONCURRENT_INFERENCES carefully for your Jetson memory budget, or scale with multiple workers (each worker keeps its own model copy).

Jetson Notes

This project requires Jetson-specific builds of PyTorch, ONNX Runtime, and cuDNN that differ from standard x86 packages. The Dockerfile handles all of this automatically, but if you need to understand or modify the build, see JETSON.md for a detailed explanation of every workaround and why it's needed.