Scalattice Modelmeter

This is an internal tool used by our engineers at Scalattice to benchmark our inference API for price and latency against rivals offering open models. We decided to release it here because we believe it can further the development of inference networks and products, and help build a world of inference abundance.

git clone https://github.com/scalattice/modelmeter.git
cd modelmeter
./run.sh
# → http://127.0.0.1:8765

./run.sh creates a venv, installs, and starts the server. Pass-through flags work: ./run.sh --host 0.0.0.0 --port 8765.

Manual equivalent:

python3 -m venv .venv
source .venv/bin/activate
pip install -e .
modelmeter

Capabilities

• Search and filter live open-model catalogs
• $/1M input & output, blend, tokens per dollar, and a simple bill estimate
• Compare / Try - same prompt (optional image, headers) against multiple endpoints; latency, tok/s, estimated cost
• Shareable filter URLs (?q=qwen3-8b&in=10e6&out=2e6)
• Per-provider API tokens and custom OpenAI-compatible hosts in Settings

Catalog sources

Credentials live in ~/.modelmeter/modelmeter.db (override with MODELMETER_DATA_DIR).

Deploy

modelmeter --host 0.0.0.0 --port 8765

Docker:

docker build -t modelmeter .
docker run --rm -p 8765:8765 modelmeter

License

MIT