Termite

ML inference service for embeddings, chunking, reranking, classification, NER, OCR, transcription, text generation, and more — with two-tier caching (memory + singleflight).

Termite is the companion ML service for Antfly, the distributed search engine. It runs automatically in Antfly's swarm mode and can also be used standalone. If you're new, the Antfly quickstart is the fastest way to see everything working together.

Documentation | Discord

Features

• Embeddings — dense and sparse vectors, multimodal (text, images, audio)
• Chunking — semantic text segmentation
• Reranking — cross-encoder relevance scoring
• Classification — zero-shot text classification (NLI-based, 100+ languages)
• Recognition (NER) — named entity recognition, zero-shot labels, relation extraction
• Reading (OCR) — document understanding, OCR, visual question answering
• Transcription — speech-to-text (Whisper, Wav2Vec2)
• Extraction — schema-based structured data extraction
• Rewriting — paraphrasing, question generation (Seq2Seq)
• Generation — text generation with tool calling (OpenAI-compatible)
• Multiple backends — ONNX Runtime, XLA (TPU/CUDA), pure Go
• SIMD / SME acceleration — vector math uses hardware intrinsics via go-highway on x86 and ARM
• Native Go ML — XLA backend powered by GoMLX and GoLLMX, working toward making native Go ML/LLM inference a reality
• Kubernetes operator — autoscaling with TermitePool and TermiteRoute CRDs

Running

# Standalone server
go run ./cmd/termite run

Inference Backends

Termite supports multiple inference backends, selected via build tags. The omni build includes everything so you can pick at runtime.

Omni Build (Recommended)

Includes both ONNX and XLA backends — pick which one to use at runtime without recompiling.

# Download dependencies for all platforms
./scripts/download-onnxruntime.sh
./scripts/download-pjrt.sh

# Build omni binary
CGO_ENABLED=1 go build -tags="onnx,ORT,xla,XLA" -o termite ./pkg/termite/cmd

# Run with backend priority (tries in order until one works)
./termite run --backend-priority="onnx:cuda,xla:tpu,onnx:cpu,go"

ONNX Runtime

Dependencies:

• ONNX Runtime - download for your platform or install with homebrew
• Tokenizers - HuggingFace tokenizers bindings

# Download dependencies
./scripts/download-onnxruntime.sh

# Or manually (macOS with homebrew)
CGO_ENABLED=1 \
DYLD_LIBRARY_PATH=/opt/homebrew/opt/onnxruntime/lib \
CGO_LDFLAGS="-L$(pwd) -ltokenizers" \
go run -tags="onnx,ORT" ./pkg/termite/cmd run

XLA Runtime (TPU/GPU)

For TPU or CUDA GPU acceleration via GoMLX XLA backend. Hardware is autodetected.

Dependencies:

• PJRT CPU Binaries - prebuilt XLA PJRT plugins
• Tokenizers - HuggingFace tokenizers bindings

# Download dependencies
./scripts/download-pjrt.sh

# Build with XLA support
go build -tags="xla,XLA" -o termite ./pkg/termite/cmd

# Run with autodetection (TPU > CUDA > CPU)
./termite run

Autodetection:

• TPU: Detected via libtpu.so, /dev/accel* devices, or GKE TPU node metadata
• CUDA: Detected via nvidia-smi or libcudart.so in library path

Installing Additional PJRT Plugins:

The omni and XLA builds bundle a CPU PJRT plugin that's auto-discovered from lib/ next to the binary. For TPU or CUDA, install the right plugin:

# Install TPU plugin (for Google Cloud TPU)
go run github.com/gomlx/go-xla/cmd/pjrt_installer@latest -plugin=tpu

# Install CUDA plugin (for NVIDIA GPU)
go run github.com/gomlx/go-xla/cmd/pjrt_installer@latest -plugin=cuda

# Install to a specific location
go run github.com/gomlx/go-xla/cmd/pjrt_installer@latest -plugin=tpu -path=/usr/local/lib/go-xla

Installed plugins are found automatically via standard go-xla search paths. To override, set PJRT_PLUGIN_LIBRARY_PATH.

Platform Availability:

Models

Pull from registry:

termite pull bge-small-en-v1.5
termite pull mxbai-rerank-base-v1
termite pull chonky-mmbert-small-multilingual-1

# List available models
termite list --remote

Models auto-discovered from chunker_models_dir, embedder_models_dir, reranker_models_dir.

Available Models

Embedders

Multimodal Embedders

Rerankers

Chunkers

Classifiers

Recognizers (NER)

Readers (OCR / Document Understanding)

Transcribers (Speech-to-Text)

Extractors

Rewriters

Generators

Model Variants

Models come in multiple precision variants, trading off size and speed for accuracy:

Pull specific variants:

# Pull using variant suffix (recommended)
termite pull bge-small-en-v1.5-i8

# Or use --variants flag
termite pull --variants i8 bge-small-en-v1.5

# Pull multiple models with same variant
termite pull bge-small-en-v1.5-i8 mxbai-rerank-base-v1-i8

# Pull multiple variants for one model
termite pull --variants f16,i8 bge-small-en-v1.5

Use variants in config:

embedder:
  provider: termite
  model: bge-small-en-v1.5-f16  # Use FP16 variant

Termite auto-selects the best available variant if not specified.

API

All endpoints accept JSON. See openapi.yaml for full schema details.

Multimodal Input

The /api/embed endpoint supports multimodal input using the OpenAI content format:

{
  "model": "clip-vit-base-patch32",
  "input": [
    {
      "content": [
        {"type": "text", "text": "a photo of a cat"},
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
      ]
    }
  ]
}

Configuration

Config via file (termite.yaml), flags, or environment variables (TERMITE_ prefix):

api_url: "http://localhost:11433"
models_dir: "./models"

# Backend priority with optional device specifiers
# Format: "backend" or "backend:device"
# Devices: auto (default), cuda, coreml, tpu, cpu
backend_priority:
  - onnx:cuda      # Try ONNX with CUDA first
  - xla:tpu        # Then XLA with TPU
  - onnx:cpu       # Fall back to ONNX CPU
  - go             # Pure Go fallback (always works)

keep_alive: "5m"
max_loaded_models: 3
allow_downloads: false
log:
  level: info
  style: terminal

Backend Priority

The backend_priority setting controls which backends Termite tries, in order. Each entry can be:

• Backend only: onnx, xla, go - uses auto device detection
• Backend with device: onnx:cuda, xla:tpu, onnx:coreml - explicit device

Available backends (depend on build tags):

Example configurations:

# GPU-first with CPU fallback
backend_priority: ["onnx:cuda", "xla:cuda", "onnx:cpu", "go"]

# macOS with CoreML acceleration
backend_priority: ["onnx:coreml", "go"]

# Cloud TPU deployment
backend_priority: ["xla:tpu", "xla:cpu"]

# Simple auto-detection (default)
backend_priority: ["onnx", "xla", "go"]

Kubernetes Operator

Deploy on GKE with TPU support using the Termite Operator.

Custom Resources

TermitePool: manages a pool of Termite replicas with autoscaling.

apiVersion: termite.antfly.io/v1alpha1
kind: TermitePool
metadata:
  name: embeddings-pool
spec:
  workloadType: read-heavy
  models:
    preload:
      - name: bge-small-en-v1.5
        variant: i8
        priority: high
        strategy: eager    # Always loaded, never evicted
      - name: mxbai-rerank-base-v1
        variant: i8
        priority: high
        # strategy defaults to loadingStrategy (lazy)
    loadingStrategy: lazy  # Default for models without explicit strategy
    keepAlive: 5m          # Idle timeout for lazy models
  replicas:
    min: 2
    max: 10
  hardware:
    accelerator: tpu-v5-lite-podslice
    topology: "2x2"
  autoscaling:
    enabled: true
    metrics:
      - type: queue-depth
        target: "50"

TermiteRoute: routes traffic to pools based on model or endpoint.

Running the Operator

# Build operator
go build -o termite-operator ./cmd/termite-operator

# Generate CRDs and RBAC manifests
make generate

See pkg/operator/ for CRD definitions and controller implementation. The model registry protocol is formally specified in TLA+.

Community

Discord for questions, discussion, and updates.

License

Apache License 2.0