model2vec-serve

Lightweight OpenAI and Text Embedding Inference (TEI) compatible embeddings server for model2vec static embedding models.

Features

• OpenAI-compatible POST /v1/embeddings and GET /v1/models
• TEI-compatible POST /embed and GET /info, plus per-model POST /tei/{model_id}/embed and GET /tei/{model_id}/info
• Optional API key authentication
• Health (/health, /ready) and metrics (/metrics) endpoints
• Interactive OpenAPI documentation at /docs
• Structured JSON logs with request correlation IDs
• Small, containerized Rust binary
• One-command two-model local deployment via Docker Compose
• Helm chart for Kubernetes deployment with volume mount support

Quickstart

Run locally with a Hugging Face model id:

cargo run --release -- --model minishlab/potion-multilingual-128M --port 8080

Request embeddings:

curl -X POST http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input":"Hello world"}'

Serve multiple models in one process:

cargo run --release -- \
  --model minishlab/potion-multilingual-128M \
  --model minishlab/potion-code-16M-v2 \
  --default-model minishlab/potion-multilingual-128M \
  --port 8080

Select a model in the request:

curl -X POST http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input":"def hello(): pass","model":"minishlab/potion-code-16M-v2"}'

TEI per-model endpoints

TEI clients serving multiple models address a model explicitly through POST /tei/{model_id}/embed and GET /tei/{model_id}/info. The {model_id} path segment is the model's --model-alias or, by default, the last segment of its identifier (e.g. minishlab/potion-code-16M-v2 → potion-code-16M-v2). Root /embed and /info continue to serve the default model.

Breaking change in 0.5.0: the ?model= query parameter on /embed and /info was removed. Migrate as follows:

Configuration

All configuration is passed as command-line arguments:

Serve over HTTPS by pointing both TLS options at mounted files — traffic is then encrypted end to end, all the way to the application inside the container:

cargo run --release -- \
  --model minishlab/potion-multilingual-128M \
  --tls-cert /etc/model2vec-serve/tls/tls.crt \
  --tls-key /etc/model2vec-serve/tls/tls.key

Container

Build locally

docker build -t model2vec-serve:latest .
docker run -p 8080:8080 -e MODEL=minishlab/potion-multilingual-128M model2vec-serve:latest

Serve multiple models via comma-separated MODEL:

docker run -p 8080:8080 \
  -e MODEL=minishlab/potion-multilingual-128M,minishlab/potion-code-16M-v2 \
  -e DEFAULT_MODEL=minishlab/potion-multilingual-128M \
  model2vec-serve:latest

Pull from GitHub Container Registry

Released images are published to GHCR:

docker pull ghcr.io/freinold/model2vec-serve:v0.6.0
docker run -p 8080:8080 -e MODEL=minishlab/potion-multilingual-128M ghcr.io/freinold/model2vec-serve:v0.6.0

See docs/deployment/docker.md for the full release and tagging strategy.

Docker Compose

Run a local two-model stack (multilingual + code v2) with a persisted model cache using the published image:

docker compose up -d

The stack serves minishlab/potion-multilingual-128M (default) and minishlab/potion-code-16M-v2. Models download once into ./models and survive restarts. Customize via .env (see .env.example).

See docs/deployment/compose.md for the full guide, including prerequisites, volume mounting, configuration, and troubleshooting.

Helm

The chart is published to the GitHub Container Registry:

helm install model2vec-serve \
  oci://ghcr.io/freinold/model2vec-serve/model2vec-serve \
  --version 0.6.0 \
  --set models[0]=minishlab/potion-multilingual-128M \
  --set apiKey=your-secret-key

Or install from a local checkout:

helm install model2vec-serve ./helm/model2vec-serve \
  --set model=minishlab/potion-multilingual-128M \
  --set apiKey=your-secret-key

Install with multiple models:

helm install model2vec-serve ./helm/model2vec-serve \
  --set models={minishlab/potion-multilingual-128M,minishlab/potion-code-16M-v2} \
  --set defaultModel=minishlab/potion-multilingual-128M \
  --set apiKey=your-secret-key

See helm/model2vec-serve/README.md for more options, including volume-mounted models.

Development

Run the test suite:

cargo test

Run linting and formatting:

cargo fmt --check
cargo clippy --all-targets --all-features -- -D warnings

Run benchmarks:

cargo bench