In order to get started, first make sure you have uv installed. The official documentation has instructions for how to install the uv CLI.

If you already have uv installed you can run make setup to install the dependencies and set up the environment. If you use conda or venv directly to manage your python environment you can install uv with pip install uv and then run make setup to install the dependencies.

Example with conda:

conda create -n <your-env-name> python=3.11
conda activate <your-env-name>
pip install uv
make setup

Alternatively, you can use uv directly to manage the environment:

# Install dependencies and create virtual environment
uv sync

# Activate the environment (if needed)
source .venv/bin/activate  # On macOS/Linux
# or
.venv\Scripts\activate     # On Windows

OpenBench provides a powerful command-line interface for easy interaction with the benchmarking framework. The CLI offers three main commands for different use cases:

Available Commands

evaluate - Run Benchmark Evaluations

Run comprehensive evaluations of your pipelines on datasets with configurable metrics.

# Evaluate using pipeline and dataset aliases
openbench-cli evaluate \
    --pipeline pyannote \
    --dataset voxconverse \
    --metrics der \
    --metrics jer \
    --use-wandb \
    --wandb-project my-evaluation

# Evaluate using a configuration file
openbench-cli evaluate \
    --evaluation-config config/my_evaluation.yaml \
    --evaluation-config-overrides wandb.project=my-project

# Get help and see available options
openbench-cli evaluate --help

inference - Run Single Audio Inference

Test your pipeline on individual audio files for quick validation.

# Run inference on a single audio file
openbench-cli inference \
    --pipeline pyannote \
    --audio-path path/to/audio.wav \
    --output-path path/to/output.json

# Run inference with custom configuration
openbench-cli inference \
    --pipeline pyannote \
    --audio-path path/to/audio.wav \
    --pipeline-config '{"min_speakers": 2, "max_speakers": 5}'

summary - Explore Available Resources

Get an overview of all available pipelines, datasets, metrics, and their compatibility.

# Show all available pipelines, datasets, and metrics
openbench-cli summary

# Show only pipelines
openbench-cli summary --disable-datasets --disable-metrics --disable-compatibility

# Show only compatibility matrix
openbench-cli summary --disable-pipelines --disable-datasets --disable-metrics

# Get detailed information
openbench-cli summary --verbose

CLI Features

• Pipeline Aliases: Use friendly names like pyannote, aws-diarization, whisperx instead of class names
• Dataset Aliases: Access datasets with simple names like voxconverse, earnings21
• Metric Selection: Choose from available metrics like der, jer, wer
• Weights & Biases Integration: Built-in support for experiment tracking
• Configuration Files: Support for Hydra-based configuration management
• Verbose Output: Detailed logging for debugging and monitoring

Environment Variables

Some pipelines require specific environment variables to be set:

# AWS Transcribe
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"

# Pyannote API
export PYANNOTE_API_KEY="your-api-key"

# SpeakerKit (contact speakerkitpro@argmaxinc.com for access)
export SPEAKERKIT_CLI_PATH="/path/to/speakerkit/cli"
export SPEAKERKIT_API_KEY="your-api-key"

# Other API-based pipelines
export PICOVOICE_API_KEY="your-api-key"
export DEEPGRAM_API_KEY="your-api-key"
export FIREWORKS_API_KEY="your-api-key"
export GLADIA_API_KEY="your-api-key"
export OPENAI_API_KEY="your-api-key"

For more details about pipeline requirements, run openbench-cli summary to see the full list of available pipelines and their descriptions.

OpenBench supports different types of pipelines (Diarization, Transcription, Orchestration, and Streaming Transcription) with specific dataset schemas for each task type.

Diarization Datasets

The benchmark suite uses several speaker diarization datasets that are stored on the HuggingFace Hub. You can find all the datasets used in our evaluation in this collection. The datasets available in the aforementioned collection are:

Additional Dataset Collections

For other pipeline types, additional dataset collections are available:

• Transcription Datasets: Speech-to-text datasets compatible with transcription pipelines
• Orchestration Datasets: Diarized speech-to-text datasets for orchestration pipelines

From these datasets voxconverse and ami are not present as download options as they were already present in the HuggingFace Hub uploaded by diarizers-community.

Note: You can use openbench-cli summary to see all available pre-registered datasets and their compatibility with different pipeline types.

Dataset Schemas

OpenBench supports different pipeline types, each requiring specific dataset schemas:

Diarization Pipeline Schema

Required columns:

• audio: Audio column containing:
  • array: Audio waveform as numpy array of shape (n_samples,)
  • sampling_rate: Sample rate as integer
• timestamps_start: List of float containing start timestamps of segments in seconds
• timestamps_end: List of float containing end timestamps of segments in seconds
• speakers: List of str containing speaker IDs for each segment

Optional columns:

• uem_timestamps: List of tuples [(start, end), ...] containing Universal Evaluation Map (UEM) timestamps for evaluation

Transcription Pipeline Schema

Required columns:

• audio: Audio column containing:
  • array: Audio waveform as numpy array of shape (n_samples,)
  • sampling_rate: Sample rate as integer
• transcript: List of strings containing the words in the transcript

Optional columns:

• word_timestamps_start: List of float containing start timestamps for each word in seconds
• word_timestamps_end: List of float containing end timestamps for each word in seconds

Orchestration Pipeline Schema

Required columns:

• audio: Audio column containing:
  • array: Audio waveform as numpy array of shape (n_samples,)
  • sampling_rate: Sample rate as integer
• transcript: List of strings containing the words in the transcript
• word_speakers: List of strings containing speaker IDs for each word

Optional columns:

• word_timestamps_start: List of float containing start timestamps for each word in seconds
• word_timestamps_end: List of float containing end timestamps for each word in seconds

Validation rules:

• word_speakers and transcript must have the same length
• If word_timestamps_start and word_timestamps_end are provided, they must have the same length as transcript

Streaming Transcription Pipeline Schema

Required columns:

• audio: Audio column containing:
  • array: Audio waveform as numpy array of shape (n_samples,)
  • sampling_rate: Sample rate as integer
• text: String containing the reference transcript

Optional columns:

• word_detail: List of dictionaries containing word-level information with start and stop timestamps in samples (will be converted to seconds)

Note: Currently, most available datasets are optimized for diarization tasks. For transcription, orchestration, and streaming transcription pipelines, you may need to prepare additional annotations or use datasets that include the required fields for each task type.

Local Datasets

OpenBench supports loading datasets from local directories, which is useful for:

• Datasets that cannot be uploaded to HuggingFace Hub due to privacy/licensing restrictions
• Custom datasets you've prepared locally
• Testing with small local datasets

Local Dataset Structure

Local datasets should follow this directory structure:

dataset_dir/
├── audio/              # Audio files named by audio_id (e.g., sample_001.wav)
├── reference/          # JSON files with same names as audio (e.g., sample_001.json)
│                       # Each JSON file contains a dict with columns matching the expected schema
│                       # Example for diarization:
│                       #   {"timestamps_start": [0.0, 5.2, ...], 
│                       #    "timestamps_end": [5.2, 10.5, ...], 
│                       #    "speakers": ["SPEAKER_00", "SPEAKER_01", ...]}
│                       # Example for transcription:
│                       #   {"transcript": ["hello", "world", ...]}
├── splits/             # Split definitions (train.txt, test.txt, validation.txt)
│   └── test.txt        # Each line contains an audio_id (without extension)
└── metadata.json       # Optional: {audio_id: {extra_info_fields}}
                        #   Example: {"sample_001": {"language": "en", "dictionary": [...]}}

Reference JSON File Format

Each reference JSON file should contain a dictionary where:

• Keys match the expected columns for the dataset type (except audio, which comes from audio files)
• Values are the data that prepare_sample() expects

For example, a diarization reference file (sample_001.json) should contain:

{
  "timestamps_start": [0.0, 5.2, 10.5],
  "timestamps_end": [5.2, 10.5, 15.8],
  "speakers": ["SPEAKER_00", "SPEAKER_01", "SPEAKER_00"],
  "uem_timestamps": [[0.0, 15.8]]  // Optional
}

A transcription reference file should contain:

{
  "transcript": ["hello", "world", "how", "are", "you"],
  "word_timestamps_start": [0.0, 0.5, 1.0, 1.5, 2.0],  // Optional
  "word_timestamps_end": [0.5, 1.0, 1.5, 2.0, 2.5],    // Optional
  "language": "en",  // Optional, goes to extra_info
  "dictionary": ["hello", "world"]  // Optional, goes to extra_info
}

Using Local Datasets

To use a local dataset, simply pass the directory path as the dataset_id in your DatasetConfig:

from openbench.dataset.dataset_base import DatasetConfig
from openbench.dataset.dataset_diarization import DiarizationDataset

# Local dataset - just use a path!
config = DatasetConfig(
    dataset_id="downloaded_datasets/earnings21",  # Path to local dataset directory
    split="test"
)

# Works exactly the same as HuggingFace datasets
dataset = DiarizationDataset.from_config(config)

The system automatically detects if dataset_id is a local directory path (by checking if it exists and is a directory) and loads it accordingly. If it's not a local path, it treats it as a HuggingFace dataset ID.

Supported Audio Formats

The local dataset loader supports common audio formats:

• .wav (preferred)
• .flac
• .mp3
• .m4a

The loader will automatically detect the audio file format by trying these extensions in order.

Downloading Datasets

If you want to reproduce the exact dataset downloads and processing, you can use our dataset downloading scripts. First, make sure you have the required dependencies installed as mentioned in the Getting Started section and also install the dataset dependencies doing uv sync --group dataset

After installing the dependencies, you can run the dataset downloading script at common/download_dataset.py. For example, to download the ICSI meetings dataset, you can run:

uv run python common/download_dataset.py --dataset icsi-meetings --hf-repo-owner <your-huggingface-username>

This will download the dataset and store locally at raw_datasets/icsi-meetings directory and upload it to the designated HuggingFace organization at <your-huggingface-username>/icsi-meetings. In case you only want to download and not push to HuggingFace, you can use the --generate-only flag.

For simplicity if you want to download all the datasets you can run:

# This will download all the datasets and store them in the raw_datasets directory
# Will not push to HuggingFace
make download-datasets

NOTE:

• For datasets requiring Hugging Face access, make sure you have your HF_TOKEN environment variable set
• For the American Life Podcast dataset, you'll need Kaggle API credentials in ~/.kaggle/kaggle.json
• For Callhome and Dihard-III you need to acquire the datasets from LDC first and then set their paths in the following env variables:
  • DIHARD_DATASET_DIR if not specified it will assume the directory lives at ~/third_dihard_challenge_eval/data
  • CALLHOME_AUDIO_ROOT if not specified it will assume the directory lives at ~/callhome/nist_recognition_evaluation/r65_8_1/sid00sg1/data
• The downloaded datasets will be stored in the raw_datasets directory (which is gitignored):

OpenBench can be used as a library to evaluate your own diarization, transcription, or orchestration pipelines. The framework supports three types of pipelines:

1. Diarization Pipeline: For speaker diarization tasks
2. Transcription Pipeline: For ASR/transcription tasks
3. Orchestration Pipeline: For combined diarization and transcription tasks

Creating Your Pipeline

1. Create a new Python file (e.g., my_pipeline.py) and implement your pipeline:

from typing import Callable

from openbench.dataset import DiarizationSample
from openbench.types import PipelineType
from openbench.pipeline.base import Pipeline, register_pipeline
from openbench.pipeline.diarization.common import DiarizationOutput, DiarizationPipelineConfig
from openbench.pipeline_prediction import DiarizationAnnotation

@register_pipeline
class MyDiarizationPipeline(Pipeline):
    _config_class = MyDiarizationConfig
    pipeline_type = PipelineType.DIARIZATION

    def build_pipeline(self) -> Callable[[dict], dict]:
        # Initialize your model/function and return a callable
        return my_diarizer_function

    def parse_input(self, input_sample: DiarizationSample) -> dict:
        # Convert DiarizationSample to your model's input format
        return {
            "waveform": input_sample.waveform,
            "sample_rate": input_sample.sample_rate
        }

    def parse_output(self, output: dict) -> DiarizationOutput:
        # Convert your model's output to DiarizationOutput
        return DiarizationOutput(prediction=annotation)

2. Create a configuration class for your pipeline:

from pydantic import Field
from openbench.pipeline.diarization.common import DiarizationPipelineConfig

class MyDiarizationConfig(DiarizationPipelineConfig):
    model_path: str = Field(..., description="Path to model weights")
    threshold: float = Field(0.5, description="Detection threshold")
    num_speakers: int | None = Field(None, description="Number of speakers (optional)")

3. Create a configuration file for your pipeline:

# my_pipeline_config.yaml
out_dir: ./my_pipeline_logs
model_path: /path/to/model
threshold: 0.5
num_speakers: null

Using Your Pipeline

The CLI is currently limited to the pre-implemented pipelines in the library. For custom pipelines, you'll need to use the library directly:

from openbench.runner import BenchmarkConfig, BenchmarkRunner, WandbConfig
from openbench.metric import MetricOptions
from openbench.dataset import DiarizationDatasetConfig

from my_pipeline import MyDiarizationPipeline, MyDiarizationConfig

# Create pipeline configuration
pipeline_config = MyDiarizationConfig(
    model_path="/path/to/model",
    threshold=0.5,
    num_speakers=None,
    out_dir="./my_pipeline_logs"
)

# Create benchmark configuration
benchmark_config = BenchmarkConfig(
    wandb_config=WandbConfig(
        project_name="my-diarization-benchmark",
        run_name="my-pipeline-evaluation",
        tags=["my-pipeline", "evaluation"],
        wandb_mode="online"  # or "offline" for local testing
    ),
    metrics={
        MetricOptions.DER: {},  # Diarization Error Rate
        MetricOptions.JER: {},  # Jaccard Error Rate
    },
    datasets={
        "voxconverse": DiarizationDatasetConfig(
            dataset_id="diarizers-community/voxconverse",
            split="test"
        )
    }
)

# Create pipeline instance
pipeline = MyDiarizationPipeline(pipeline_config)

# Create and run benchmark
runner = BenchmarkRunner(benchmark_config, [pipeline])
benchmark_result = runner.run()

print(benchmark_result.global_results[0])

2. For parallel processing, you can configure the number of worker processes in your pipeline config:

pipeline_config = MyDiarizationConfig(
    model_path="/path/to/model",
    threshold=0.5,
    num_speakers=None,
    out_dir="./my_pipeline_logs",
    num_worker_processes=4,  # Number of parallel workers
    per_worker_chunk_size=2  # Samples per worker
)

3. To use Weights & Biases for experiment tracking, make sure to:
   • Set up your W&B account and get your API key
   • Make sure you're logged into your W&B account otherwise run wandb login
   • Configure the wandb_config in your benchmark configuration

The BenchmarkRunner will automatically:

• Run your pipeline on the specified datasets
• Calculate metrics for each sample
• Aggregate results globally
• Log everything to Weights & Biases (if configured)
• Handle parallel processing if enabled (specially interesting for APIs)
• Generate detailed reports and artifacts

Pipeline Types and Requirements

Diarization Pipeline

• Must implement build_pipeline(), parse_input(), and parse_output()
• Input parsing should convert DiarizationSample to your model's expected format
• Output parsing should return a DiarizationOutput with a prediction field

Transcription Pipeline

• Must implement build_pipeline(), parse_input(), and parse_output()
• Input parsing should convert DiarizationSample to your model's expected format
• Output parsing should return a TranscriptionOutput with a prediction field

Orchestration Pipeline

• Must implement build_pipeline(), parse_input(), and parse_output()
• Can either:
  • Implement end-to-end diarization and transcription
  • Use PostInferenceMergePipeline to combine separate diarization and transcription pipelines
• Output parsing should return an OrchestrationOutput with a prediction field and optionaly diarization and transcription results

The benchmark suite uses Hydra for configuration management, providing a flexible and modular way to configure evaluation runs. The configuration files are organized in the following structure:

config
├── evaluation_config.yaml                      # Main evaluation configuration
├── benchmark_config                            # Base configurations for benchmarking
│   ├── datasets                                # Dataset-specific configs
│   ├── wandb_config                            # Weights & Biases logging configs
│   └── base.yaml                               # Default benchmark_config used in evaluation_config.yaml
└── pipeline_configs                            # Predefined pipeline configurations for ease of use
    ├── my_pipeline
    │   ├── base.yaml                           # Default config used in my_pipeline.yaml
    │   └── config
    │       ├── base.yaml                       # Default config used in MyPipeline
    │       └── diarization_config
    │           ├── chunking_config             # Defines different useful chunking configurations
    │           ├── cluster_definition          # Defines different useful cluster definitions
    │           ├── speaker_embedder_config     # Defines different useful speaker embedder configurations
    │           ├── speaker_segmenter_config    # Defines different useful speaker segmenter configurations
    │           └── base.yaml                   # Default diarization_config used in evaluation_config.yaml
    ├── my_pipeline.yaml                        # Uses MyPipeline as default pipeline
    └── pyannote.yaml                           # Defines configuration for PyAnnotePipeline

Running Evaluations with Different Configurations

You can easily customize your evaluation runs using Hydra's override syntax. Here are some common usage patterns:

Using the CLI

All Hydra configuration features work with the CLI using --evaluation-config and --evaluation-config-overrides:

# Run evaluation with a specific config file
openbench-cli evaluate --evaluation-config config/my_evaluation.yaml

# Override configuration parameters
openbench-cli evaluate \
    --evaluation-config config/my_evaluation.yaml \
    --evaluation-config-overrides wandb.project=my-project pipeline_configs.MyPipeline.config.threshold=0.7

# See the resulting configuration
openbench-cli evaluate --evaluation-config config/my_evaluation.yaml --help

Using the evaluation.py script provided in the repo (old-way)

1. Selecting Specific Pipelines

# Run evaluation with only MyPipeline
uv run python evaluation.py pipeline_configs=my_pipeline

2. Modifying Pipeline Parameters You can override specific configuration parameters in two ways:

a. Override by Value:

# Change the speaker segmenter stride
uv run python evaluation.py \
    pipeline_configs=my_pipeline \
    pipeline_configs.MyPipeline.config.diarization_config.speaker_segmenter_config.variant_name=stride_2

b. Override by Config:

# Use a predefined speaker segmenter configuration
uv run python evaluation.py \
    pipeline_configs=my_pipeline \
    pipeline_configs/MyPipeline/config/diarization_config/speaker_segmenter_config=stride_2

Note: Use -h flag with any command to see the resulting configuration:

uv run python evaluation.py pipeline_configs=my_pipeline -h