All benchmarks are wrong, some will cost you less than others.

Optimum-Benchmark 🏋️

Optimum-Benchmark is a unified multi-backend & multi-device utility for benchmarking Transformers, Diffusers, PEFT, TIMM and Optimum libraries, along with all their supported optimizations & quantization schemes, for inference & training, in distributed & non-distributed settings, in the most correct, efficient and scalable way possible.

News 📰

• LlamaCpp backend for benchmarking llama-cpp-python bindings with all its supported devices 🚀
• 🥳 PyPI package is now available for installation: pip install optimum-benchmark 🎉 check it out !
• Model loading latency/memory/energy tracking for all backends in the inference scenario 🚀
• numactl support for Process and Torchrun launchers to control the NUMA nodes on which the benchmark runs.
• 4 minimal docker images (cpu, cuda, rocm) in packages for testing, benchmarking and reproducibility 🐳
• vLLM backend for benchmarking vLLM's inference engine 🚀
• Hosting the codebase of the LLM-Perf Leaderboard 🥇
• Py-TXI backend for benchmarking Py-TXI 🚀
• Python API for running isolated and distributed benchmarks with Python scripts 🐍
• Simpler CLI interface for running benchmarks (runs and sweeps) using the Hydra 🧪

Motivations 🎯

• HuggingFace hardware partners wanting to know how their hardware performs compared to another hardware on the same models.
• HuggingFace ecosystem users wanting to know how their chosen model performs in terms of latency, throughput, memory usage, energy consumption, etc compared to another model.
• Benchmarking hardware & backend specific optimizations & quantization schemes that can be applied to models and improve their computational/memory/energy efficiency.

[!Note] Optimum-Benchmark is a work in progress and is not yet ready for production use, but we're working hard to make it so. Please keep an eye on the project and help us improve it and make it more useful for the community. We're looking forward to your feedback and contributions. 🚀  

CI Status 🚦

Optimum-Benchmark is continuously and intensively tested on a variety of devices, backends, scenarios and launchers to ensure its stability with over 300 tests running on every PR (you can request more tests if you want to).

API 📈

CLI 📈

Quickstart 🚀

Installation 📥

Using uv (Recommended)

uv is a fast Python package manager that we recommend for installing and managing dependencies:

# Install uv if you haven't already
pip install uv

# Add optimum-benchmark to your uv project
uv add optimum-benchmark

# Or run optimum-benchmark with uv as a command without installing it 
# (automaticlly installs and runs it in an isolated virtual environment)
uv run optimum-benchmark --config-dir examples/ --config-name cuda_pytorch_bert

# Or run optimum-benchmark with uv script without installing it
# (automaticlly installs and runs it in an isolated virtual environment)
uv run script.py
script.py:
# /// script
# dependencies = [
#   "optimum-benchmark",
# ]
# ///
# [...]

# Or clone the repository and create an environment with optimum-benchmark installed for development
git clone https://github.com/huggingface/optimum-benchmark.git
cd optimum-benchmark
uv sync

Using pip

You can also install using traditional pip:

pip install optimum-benchmark

or install the latest version from the main branch:

pip install git+https://github.com/huggingface/optimum-benchmark.git

or if you want to tinker with the code, you can clone the repository and install it in editable mode:

git clone https://github.com/huggingface/optimum-benchmark.git
cd optimum-benchmark
pip install -e .

Running benchmarks using the Python API 🧪

You can run benchmarks from the Python API as simple python scripts. This is especially cool when used with uv scripts. Here's an example of how to run an isolated benchmark using the pytorch backend, torchrun launcher and inference scenario with latency and memory tracking enabled.

# /// script
# dependencies = [
#   "optimum-benchmark",
# ]
# ///

from optimum_benchmark import Benchmark, BenchmarkConfig, TorchrunConfig, InferenceConfig, PyTorchConfig
from optimum_benchmark.logging_utils import setup_logging

setup_logging(level="INFO")

if __name__ == "__main__":
    launcher_config = TorchrunConfig(nproc_per_node=2)
    scenario_config = InferenceConfig(latency=True, memory=True)
    backend_config = PyTorchConfig(model="gpt2", device="cuda", device_ids="0,1", no_weights=True)
    benchmark_config = BenchmarkConfig(
        name="pytorch_gpt2",
        scenario=scenario_config,
        launcher=launcher_config,
        backend=backend_config,
    )
    benchmark_report = Benchmark.launch(benchmark_config)
    # push artifacts to the hub
    benchmark_config.push_to_hub("IlyasMoutawwakil/pytorch_gpt2") # or benchmark_config.save_json("pytorch_gpt2/benchmark_config.json")
    benchmark_report.push_to_hub("IlyasMoutawwakil/pytorch_gpt2") # or benchmark_report.save_json("pytorch_gpt2/benchmark_report.json")

    # merge them into a single artifact and push to the hub
    benchmark = Benchmark(config=benchmark_config, report=benchmark_report)
    benchmark.push_to_hub("IlyasMoutawwakil/pytorch_gpt2") # or benchmark.save_json("pytorch_gpt2/benchmark.json")

Running the above with uv will automatically runs it in an isolated virtual environment with optimum-benchmark installed, and it's as simple as:

uv run script.py

You can also see the available parameters of different configuration classes in the Features section below.

Running benchmarks using the Hydra CLI 🧪

You can also run a benchmark using the command line by specifying the configuration directory and the configuration name. Both arguments are mandatory for hydra. --config-dir is the directory where the configuration files are stored and --config-name is the name of the configuration file without its .yaml extension.

optimum-benchmark --config-dir examples/ --config-name cuda_pytorch_bert

This will run the benchmark using the configuration in examples/cuda_pytorch_bert.yaml and store the results in runs/cuda_pytorch_bert.

The resulting files are :

• benchmark_config.json which contains the configuration used for the benchmark, including the backend, launcher, scenario and the environment in which the benchmark was run.
• benchmark_report.json which contains a full report of the benchmark's results, like latency measurements, memory usage, energy consumption, etc.
• benchmark_report.txt which contains a detailed report of the benchmark's results, in the same format they were logged.
• benchmark_report.md which contains a detailed report of the benchmark's results, in markdown format.
• benchmark.json contains both the report and the configuration in a single file.
• benchmark.log contains the logs of the benchmark run.

optimum-benchmark --config-dir examples/ --config-name pytorch_bert backend.model=gpt2 backend.device=cuda

optimum-benchmark --config-dir examples --config-name pytorch_bert -m backend.device=cpu,cuda

Features 🎨

optimum-benchmark allows you to run benchmarks with minimal configuration. A benchmark is defined by three main components:

• The launcher to use (e.g. process)
• The scenario to follow (e.g. training)
• The backend to run on (e.g. onnxruntime)

Launchers 🚀

• Process launcher (launcher=process); Launches the benchmark in an isolated process.
• Torchrun launcher (launcher=torchrun); Launches the benchmark in multiples processes using torch.distributed.
• Inline launcher (launcher=inline), not recommended for benchmarking, only for debugging purposes.

Scenarios 🏋

• Training scenario (scenario=training) which benchmarks the model using the trainer class with a randomly generated dataset.
• Inference scenario (scenario=inference) which benchmakrs the model's inference method (forward/call/generate) with randomly generated inputs.

Backends & Devices 📱

• PyTorch backend for CPU (backend=pytorch, backend.device=cpu)
• PyTorch backend for CUDA (backend=pytorch, backend.device=cuda, backend.device_ids=0,1)
• PyTorch backend for Habana Gaudi Processor (backend=pytorch, backend.device=hpu, backend.device_ids=0,1)
• ONNXRuntime backend for CPUExecutionProvider (backend=onnxruntime, backend.device=cpu)
• ONNXRuntime backend for CUDAExecutionProvider (backend=onnxruntime, backend.device=cuda)
• ONNXRuntime backend for ROCMExecutionProvider (backend=onnxruntime, backend.device=cuda, backend.provider=ROCMExecutionProvider)
• ONNXRuntime backend for TensorrtExecutionProvider (backend=onnxruntime, backend.device=cuda, backend.provider=TensorrtExecutionProvider)
• Py-TXI backend for CPU and GPU (backend=py-txi, backend.device=cpu or backend.device=cuda)
• Neural Compressor backend for CPU (backend=neural-compressor, backend.device=cpu)
• TensorRT-LLM backend for CUDA (backend=tensorrt-llm, backend.device=cuda)
• OpenVINO backend for CPU (backend=openvino, backend.device=cpu)
• OpenVINO backend for GPU (backend=openvino, backend.device=gpu)
• vLLM backend for CUDA (backend=vllm, backend.device=cuda)
• vLLM backend for ROCM (backend=vllm, backend.device=rocm)
• vLLM backend for CPU (backend=vllm, backend.device=cpu)
• IPEX backend for CPU (backend=ipex, backend.device=cpu)
• IPEX backend for XPU (backend=ipex, backend.device=xpu)

Contributing 🤝

Contributions are welcome! And we're happy to help you get started. Feel free to open an issue or a pull request. Things that we'd like to see:

• More backends (Tensorflow, TFLite, Jax, etc).
• More tests (for optimizations and quantization schemes).
• More hardware support (Habana Gaudi Processor (HPU), Apple M series, etc).
• Task evaluators for the most common tasks (would be great for output regression).

To get started, you can check the CONTRIBUTING.md file.