WhisperX ASR API Service

⚠️ Alpha Version - For Self-Hosting Enthusiasts

A simple ASR API service powered by WhisperX for transcription with speaker diarization. Built for self-hosters running Speakr or similar applications.

Two Docker image variants are published per release.

Device	Pull this tag
RTX 50xx GPU (Blackwell)	learnedmachine/whisperx-asr-service:blackwell (PyTorch 2.8.0 / cu128)
CPU or any other NVIDIA GPU (10xx, 20xx, 30xx, 40xx, A-series, H-series)	learnedmachine/whisperx-asr-service:latest (PyTorch 2.7.1 / cu126)

See Image Variants for details.

What This Does

• Transcribes audio files using OpenAI Whisper models
• Identifies speakers ("Who spoke when") using Pyannote.audio
• Returns word-level timestamps
• Supports 90+ languages
• Outputs JSON, SRT, VTT, TSV formats
• Runs on your own GPU hardware via Docker

Limitations

• Not production-grade: Basic error handling, no authentication
• GPU required: Needs an NVIDIA GPU; ~5GB VRAM covers the large models
• File size limits: Large audio files (>1GB) can cause out-of-memory errors
• VRAM usage: Memory consumption increases with file size and diarization
• Alpha software: Expect bugs and breaking changes

How It Works

Audio --> Whisper (transcription) --> Wav2Vec2 (alignment) --> Pyannote (speaker ID) --> Output

The service supports two serving modes:

• Simple mode (default): Single-process uvicorn with an async GPU queue. Requests are serialized through a semaphore so only one pipeline runs on the GPU at a time. Good for single-GPU, low-traffic, or development use.
• Ray Serve mode: Runs on Ray Serve with cross-request batching (@serve.batch). Scales from 1 GPU to multi-GPU. Two pipeline strategies are available:
  • Replicate (default): Each GPU runs the complete pipeline (Whisper + Align + Diarize). 4 GPUs = 4 independent pipeline replicas, no cross-GPU data transfer.
  • Split: Each stage runs as a separate deployment with fractional GPU allocation. Useful when you want independent scaling per stage.

Prerequisites

Hardware Requirements

GPU memory requirements vary by model size:

Note: Measured on an RTX 3090 (float16, batch 16) running transcription + alignment + pyannote community-1 diarization. Leave headroom above these figures for concurrent requests and larger batches.

Minimum Configuration (small/medium models):

• GPU: NVIDIA GPU with 6GB VRAM (e.g. RTX 2060) or better
• CPU: 8+ cores
• RAM: 16GB
• Storage: 50GB SSD

Recommended (large-v3 with diarization):

• GPU: NVIDIA RTX 3090 (24GB VRAM) or RTX 4090
• CPU: 12+ cores
• RAM: 32GB
• Storage: 100GB SSD

Software Requirements

• Docker and Docker Compose
• NVIDIA Docker Runtime (for GPU support)
• Hugging Face Account (for speaker diarization models)

Image Variants

Three prebuilt Docker images are published per release. They differ in the PyTorch wheel and platform they ship; the application code is identical.

If you have an RTX 50xx, use the -blackwell tag. On a DGX Spark, use the -spark tag. Everyone else: use :latest.

Tested hardware disclaimer. I develop and test on RTX 3090 (Ampere) hardware only. The default :latest images are validated end to end on that hardware before release. The -blackwell and -spark variants are built from the same source in CI but I cannot run them; they rely on community validation. The Spark recipe was contributed and hardware-tested by the community (PR #23). Please report regressions on these variants in the issues.

To build a custom variant locally, override the build args:

docker build \
  --build-arg TORCH_VERSION=2.7.1 \
  --build-arg TORCH_INDEX_URL=https://download.pytorch.org/whl/cu126 \
  -t whisperx-asr-service:custom .

Quick Start (Prebuilt Image)

Get up and running in 3 steps using the prebuilt Docker image:

1. Get Hugging Face Token and Model Access

Speaker diarization requires a Hugging Face token and model access:

a) Create Hugging Face Account:

• Visit: https://huggingface.co/join and sign up

b) Accept Model User Agreements (ALL REQUIRED):

You need to accept agreements for all three models:

1. pyannote/speaker-diarization-community-1 - Click "Agree and access repository"
2. pyannote/segmentation-3.0 - Click "Agree and access repository"
3. pyannote/speaker-diarization-3.1 - Click "Agree and access repository"

c) Generate Access Token:

• Visit: https://huggingface.co/settings/tokens
• Click "New token", name it (e.g., "whisperx-diarization")
• Select "Read" permission and generate
• Copy the token (starts with hf_...)

⚠️ Important: Without accepting all model agreements, you'll get "403 Access Denied" errors.

2. Create Configuration File

Create a .env file with your Hugging Face token:

# Create .env file
cat > .env << 'EOF'
HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
DEVICE=cuda
COMPUTE_TYPE=float16
BATCH_SIZE=16
PRELOAD_MODEL=large-v3
MAX_FILE_SIZE_MB=1000
# Serve mode: simple (default) or ray (Ray Serve with batching)
SERVE_MODE=simple
EOF

Replace hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx with your actual token.

3. Run with Docker Compose (Recommended)

Download the docker-compose.yml file and start the service:

# Download docker-compose.yml
curl -O https://raw.githubusercontent.com/murtaza-nasir/whisperx-asr-service/main/docker-compose.yml

# RTX 50xx (Blackwell) only: switch to the blackwell image
# sed -i 's|whisperx-asr-service:latest|whisperx-asr-service:blackwell|' docker-compose.yml

# Start the service (pulls prebuilt image automatically)
docker compose up -d

# Check logs
docker compose logs -f

The bundled docker-compose.yml uses :latest (PyTorch 2.7.1 / cu126), which works on every NVIDIA card from Pascal through Hopper. RTX 50xx users should swap to :blackwell (PyTorch 2.8.0 / cu128). See Image Variants for the full matrix.

Or run with Docker command:

# Pick the right tag for your GPU:
IMAGE=learnedmachine/whisperx-asr-service:latest      # 10xx-40xx, A/H-series
# IMAGE=learnedmachine/whisperx-asr-service:blackwell # RTX 50xx only

docker run -d \
  --name whisperx-asr-api \
  --gpus all \
  -p 9000:9000 \
  -e DEVICE=cuda \
  -e COMPUTE_TYPE=float16 \
  -e BATCH_SIZE=16 \
  -e HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
  -e PRELOAD_MODEL=large-v3 \
  -v whisperx-cache:/.cache \
  --restart unless-stopped \
  "$IMAGE"

The service will be available at http://localhost:9000

4. Test the Service

# Health check
curl http://localhost:9000/health

# Test transcription
curl -X POST http://localhost:9000/asr \
  -F "audio_file=@your_audio.mp3" \
  -F "language=en"

---

Build from Source (Advanced)

For development or if you want to build from source:

1. Clone Repository

git clone https://github.com/murtaza-nasir/whisperx-asr-service.git
cd whisperx-asr-service

2. Set Up Environment

# Copy example environment file
cp .env.example .env

# Edit .env and add your Hugging Face token
nano .env

3. Build and Run

Using docker-compose.dev.yml (with live code mounting):

# Build and start
docker compose -f docker-compose.dev.yml up -d --build

# Check logs
docker compose -f docker-compose.dev.yml logs -f

Or build manually:

# Build image
docker build -t whisperx-asr-service .

# Run container
docker run -d \
  --name whisperx-asr-api \
  --gpus all \
  -p 9000:9000 \
  --env-file .env \
  -v whisperx-cache:/.cache \
  whisperx-asr-service

Note: The docker-compose.dev.yml file mounts ./app directory for live code changes without rebuilding.

---

API Documentation

Once running, visit http://localhost:9000/docs for interactive API documentation.

Main Endpoint: POST /asr

Parameters:

Example Request (JSON output):

curl -X POST http://localhost:9000/asr \
  -F "audio_file=@meeting.mp3" \
  -F "language=en" \
  -F "model=large-v3" \
  -F "output_format=json" \
  -F "diarize=true" \
  -F "min_speakers=2" \
  -F "max_speakers=5"

Example Request (SRT subtitles):

curl -X POST http://localhost:9000/asr \
  -F "audio_file=@video.mp4" \
  -F "language=en" \
  -F "output_format=srt" \
  -F "diarize=false"

Example Response (JSON):

{
  "text": [
    {
      "start": 0.5,
      "end": 2.3,
      "text": " Hello, welcome to the meeting.",
      "speaker": "SPEAKER_00",
      "words": [
        {"word": "Hello", "start": 0.5, "end": 0.8, "score": 0.95},
        {"word": "welcome", "start": 0.9, "end": 1.2, "score": 0.93}
      ]
    }
  ],
  "language": "en",
  "segments": [...],
  "word_segments": [...]
}

Advanced Speaker Diarization Features

Exact Speaker Count

When you know the exact number of speakers, use num_speakers for more accurate diarization:

curl -X POST http://localhost:9000/asr \
  -F "audio_file=@interview.mp3" \
  -F "num_speakers=2" \
  -F "diarize=true"

This overrides min_speakers and max_speakers and typically provides better accuracy than range-based detection.

Exclusive Speaker Diarization

This service automatically uses exclusive speaker diarization when available from the pyannote community-1 model. This feature simplifies reconciliation between fine-grained speaker diarization timestamps and transcription timestamps, making it ideal for applications like Speakr where you need to align transcripts with speaker segments.

Benefits:

• More accurate timestamp alignment between speakers and words
• Better handling of speaker transitions
• Simplified post-processing for multi-speaker transcripts

Tuning Diarization Hyperparameters

When speakers are merged into a single label, or short back-and-forth turns are missed, the pyannote community-1 pipeline can be tuned through environment variables. These are unset by default, so the service runs with the model's published defaults unless you opt in. The pipeline's full default parameter schema is logged at startup; for community-1 it is {'segmentation': {'min_duration_off': 0.0}, 'clustering': {'threshold': 0.6, 'Fa': 0.07, 'Fb': 0.8}}.

# Split merged speakers (the most common fix); tag any orphan segments
DIARIZE_CLUSTERING_THRESHOLD=0.5
DIARIZE_FILL_NEAREST=true

These are global settings applied when the pipeline loads. Invalid or unrecognised keys are logged and ignored rather than failing diarization, so the pipeline always falls back to defaults if an override cannot be applied. To find the best value for your audio without restarting the service, use the sweep script described below.

Sweeping Parameters on Your Own Audio

tests/diarize_sweep.py runs diarization on one file across several settings and reports, per config, the number of speakers, turns, and a turn timeline, so you can see which value recovers your missing turns. It isolates the diarizer (no transcription), so it is fast. Run it as a one-off container so it does not disturb a running service:

docker run --rm --gpus '"device=0"' \
  -v whisperx-cache:/.cache \
  -v "$PWD/tests:/work/tests:ro" -v "$PWD/testfiles:/work/testfiles:ro" \
  -e HF_TOKEN="$HF_TOKEN" -e HF_HOME=/.cache -e DEVICE=cuda \
  learnedmachine/whisperx-asr-service:latest \
  python3 /work/tests/diarize_sweep.py /work/testfiles/your_audio.mp3 --min-speakers 2

Pass --grid "none:none,0.5:0.0,0.45:0.0" for a custom set of threshold:min_duration_off pairs, and --num-speakers / --min-speakers / --max-speakers when the count is known.

For audio that mixes languages within a single file (for example English and Mandarin in one meeting), also consider passing an explicit language per request: WhisperX loads one alignment model for the detected language, and poor word-level timestamps from a mismatched alignment model are a common cause of merged or missing speaker turns. Diarization quality is also inherently limited when voices are very similar (for example synthetic or dubbed dialogue).

Custom Vocabulary (Hotwords)

Whisper often misspells brand names, acronyms, and domain-specific terms. You can improve accuracy using hotwords and initial_prompt:

• hotwords biases the model's beam search to favor specific words during decoding
• initial_prompt provides a sentence of context that primes the model to expect certain spellings

Example: transcribing audio that says "We deployed Speakr on a Kubernetes cluster using CTranslate2 for inference. PyAnnote handles the diarization."

# Baseline (no hints)
curl -X POST "http://localhost:9000/asr?language=en" \
  -F "audio_file=@meeting.mp3"
# Result: "We deployed Speaker...using ctranslate2...PnNote handles the diarization."

# With hotwords
curl -X POST "http://localhost:9000/asr?language=en&hotwords=Speakr,CTranslate2,PyAnnote" \
  -F "audio_file=@meeting.mp3"
# Result: "We deployed Speaker...using CTranslate2...PyAnnote handles the diarization."

# With hotwords + initial_prompt (best results)
curl -X POST "http://localhost:9000/asr?language=en&hotwords=Speakr,CTranslate2,PyAnnote&initial_prompt=Speakr is a transcription app." \
  -F "audio_file=@meeting.mp3"
# Result: "We deployed Speakr...using CTranslate2...PyAnnote handles the diarization."

hotwords alone fixes most spelling issues. Words that sound identical to common English words (like "Speakr" vs "Speaker") may need initial_prompt as well to provide enough context for the model to override its default.

The OpenAI-compatible endpoints (/v1/audio/transcriptions) also support a hotwords form field. If only prompt is provided, it is used as hotwords.

A test script is included to verify hotwords with your own audio:

./tests/test_hotwords.sh testfiles/your_audio.flac

# Custom hotwords
HOTWORDS="MyBrand,TechTerm" INITIAL_PROMPT="MyBrand is a product." \
  ./tests/test_hotwords.sh testfiles/your_audio.flac

Integration with Speakr

To use this service with Speakr instead of the default ASR endpoint:

If Running on the Same Machine

Update Speakr's .env file:

# Enable ASR endpoint
USE_ASR_ENDPOINT=true

# Point to WhisperX service
ASR_BASE_URL=http://whisperx-asr-api:9000

If Speakr and WhisperX are in the same Docker Compose stack, use the container name. Otherwise, use http://localhost:9000.

If Running on a Different GPU Machine

1. On GPU Machine: Deploy this service

# Make service accessible from network
# Edit docker compose.yml ports:
ports:
  - "0.0.0.0:9000:9000"  # Expose to network

2. On Speakr Machine: Update configuration

# In Speakr's .env file
USE_ASR_ENDPOINT=true
ASR_BASE_URL=http://<GPU_MACHINE_IP>:9000

Note: Replace <GPU_MACHINE_IP> with your GPU server's IP address. Use firewall rules to restrict access to trusted machines only.

Configuration

Environment Variables

Edit .env to customize:

# GPU or CPU processing
DEVICE=cuda              # cuda for GPU, cpu for CPU-only

# Computation precision
COMPUTE_TYPE=float16     # float16 (GPU), float32 (CPU), int8 (faster, lower quality)

# Alignment stage device (defaults to DEVICE). cpu keeps the Wav2Vec2
# alignment model off the GPU to reduce VRAM on small cards, at the cost
# of slower word timestamps.
# ALIGN_DEVICE=cpu

# Batch size (higher = faster but more memory). Default is device-aware:
# 16 on cuda, 2 on cpu. Long audio on CPU benefits from BATCH_SIZE=1.
BATCH_SIZE=16           # 16 for 8GB VRAM, 32+ for high-end GPUs, 1-2 on CPU

# Hugging Face token for diarization
HF_TOKEN=hf_xxx...

# Model preloading (optional, reduces first-request latency)
PRELOAD_MODEL=large-v3   # Leave empty to disable, or set to: tiny, base, small, medium, large-v2, large-v3

# Maximum file size in MB (prevents out-of-memory errors)
MAX_FILE_SIZE_MB=1000    # Default 1GB, adjust lower for GPUs with <16GB VRAM

# Idle model eviction (default disabled). When > 0, Whisper models that have
# not served a request in this many seconds are unloaded from memory by a
# background sweep. The next request that needs the model will reload it.
MODEL_KEEP_ALIVE_SECONDS=0          # 0 disables eviction; e.g. 3600 = 1 hour
MODEL_EVICTION_INTERVAL_SECONDS=60  # Sweep frequency (floor of 30 seconds)

# Diarization hyperparameter tuning (optional, unset = model defaults).
# See "Tuning Diarization Hyperparameters" above for details.
# DIARIZE_CLUSTERING_THRESHOLD=0.5   # default 0.6; lower = split merged voices
# DIARIZE_MIN_DURATION_OFF=0.2       # default 0.0; raise = reduce over-segmentation
# DIARIZE_PARAM_OVERRIDES={"clustering": {"Fb": 1.0}}  # JSON escape hatch (Fa/Fb)
# DIARIZE_FILL_NEAREST=false         # true = tag orphan segments with nearest speaker

# Rebuild segments at speaker-change boundaries after diarization, so rapid
# turns are not merged into one speaker's segment. Always on for the qwen3
# backend; opt-in here for the whisper backend (changes segment shape
# compared to previous releases).
# RESEGMENT_BY_SPEAKER=true

# Experimental Qwen3-ASR backend (see "Qwen3-ASR Backend" below).
# ASR_BACKEND=qwen3
# QWEN3_ASR_MODEL=Qwen/Qwen3-ASR-1.7B-hf
# QWEN3_ALIGNER_MODEL=Qwen/Qwen3-ForcedAligner-0.6B-hf
# QWEN3_CHUNK_SECONDS=90
# QWEN3_DEFAULT_CONTEXT=            # standing vocabulary/context biasing text

Serve Mode

Controlled by the SERVE_MODE environment variable in your .env file.

Simple Mode (default)

SERVE_MODE=simple

Runs uvicorn directly. Requests are serialized through an async GPU semaphore so the event loop stays responsive while GPU work runs in a thread pool. This is backward-compatible with previous versions.

You can tune GPU_CONCURRENCY=1 (default) to control how many pipeline runs execute concurrently. Leave at 1 for single-GPU setups.

Ray Serve Mode

SERVE_MODE=ray

Runs on Ray Serve with cross-request batching (@serve.batch). Two pipeline strategies are available, controlled by PIPELINE_STRATEGY:

Replicate Strategy (default, recommended)

PIPELINE_STRATEGY=replicate
NUM_GPU_REPLICAS=4       # one full pipeline per GPU

Each GPU runs the complete 3-stage pipeline. Ray Serve routes incoming requests across replicas.

                    +-- GPU 0: [Whisper + Align + Diarize] --+
HTTP --> Proxy -->  +-- GPU 1: [Whisper + Align + Diarize] --+--> Response
                    +-- GPU 2: [Whisper + Align + Diarize] --+
                    +-- GPU 3: [Whisper + Align + Diarize] --+

Split Strategy

PIPELINE_STRATEGY=split

Each pipeline stage runs as a separate deployment with fractional GPU allocation and cross-request batching (@serve.batch). Stage-level pipelining means request B can start transcription while request A is still in diarization.

HTTP --> Proxy --> ASR Ingress
                       |
              +--------+--------+
              |        |        |
          Whisper   Align    Diarize
         (GPU 0.5) (GPU 0.3) (GPU 0.2)

Strategy Comparison

Both strategies achieve similar throughput (~6.3-6.5x speedup on 4x RTX 3090 with 8 concurrent workers).

Ray Serve Configuration

All optional, shown with defaults:

# Pipeline strategy: replicate (full pipeline per GPU) or split (stage per GPU)
PIPELINE_STRATEGY=replicate

# Number of pipeline replicas (set to number of GPUs)
NUM_GPU_REPLICAS=1

# Cross-request batch sizes per stage (tune for GPU VRAM)
WHISPER_BATCH_SIZE=4
ALIGN_BATCH_SIZE=8
DIARIZE_BATCH_SIZE=2

# Seconds to wait collecting a batch before processing what's available
BATCH_WAIT_TIMEOUT=0.1

# Split strategy only: fractional GPU allocation per stage
WHISPER_GPU_FRACTION=0.5
ALIGN_GPU_FRACTION=0.3
DIARIZE_GPU_FRACTION=0.2

# Split strategy only: per-stage replica overrides (fall back to NUM_GPU_REPLICAS)
# WHISPER_NUM_REPLICAS=2
# ALIGN_NUM_REPLICAS=1
# DIARIZE_NUM_REPLICAS=1

Multi-GPU Examples

# 4 GPUs, full pipeline on each (recommended)
SERVE_MODE=ray
PIPELINE_STRATEGY=replicate
NUM_GPU_REPLICAS=4

# 3 GPUs, one stage per GPU
SERVE_MODE=ray
PIPELINE_STRATEGY=split
WHISPER_GPU_FRACTION=1.0
ALIGN_GPU_FRACTION=1.0
DIARIZE_GPU_FRACTION=1.0

# 4 GPUs, hybrid: 2 Whisper + 1 Align + 1 Diarize
SERVE_MODE=ray
PIPELINE_STRATEGY=split
WHISPER_NUM_REPLICAS=2  WHISPER_GPU_FRACTION=1.0
ALIGN_NUM_REPLICAS=1    ALIGN_GPU_FRACTION=1.0
DIARIZE_NUM_REPLICAS=1  DIARIZE_GPU_FRACTION=1.0

When running in Ray mode, the Ray Dashboard is available at http://localhost:8265 for monitoring deployments, replicas, and request metrics.

GPU Pinning

To restrict which GPUs the service uses, create a docker-compose.dev.local.yml override (gitignored):

services:
  whisperx-asr:
    environment:
      - CUDA_VISIBLE_DEVICES=0,1,2,3
      - NUM_GPU_REPLICAS=4

Use CUDA_VISIBLE_DEVICES (not NVIDIA_VISIBLE_DEVICES) since the Docker Compose deploy section exposes all GPUs to the container.

Then start with:

docker compose -f docker-compose.dev.yml -f docker-compose.dev.local.yml up -d

Model Selection

Available Whisper models (speed vs accuracy tradeoff):

*Measured on an RTX 3090 (float16, BATCH_SIZE=16, 12-minute file), running transcription, alignment, and pyannote diarization. Peak VRAM occurs during transcription; alignment and diarization add little on top. Actual peaks vary somewhat with audio content and batch size.

Recommendation:

• Use large-v3 for best quality (fits comfortably in 6GB+ VRAM)
• Use distil-large-v3.5 or large-v3-turbo for near-large quality at about half the VRAM and higher speed
• Use small or medium for tight VRAM budgets (2-3GB)

Qwen3-ASR Backend (Experimental)

Setting ASR_BACKEND=qwen3 replaces the transcription and alignment stages with Qwen3-ASR (Qwen/Qwen3-ASR-1.7B-hf) and the Qwen3 forced aligner (Qwen/Qwen3-ForcedAligner-0.6B-hf), loaded through stock transformers. Diarization is unchanged.

Why use it:

• The forced aligner is a single model covering zh, en, yue, fr, de, it, ja, ko, pt, ru, and es. Code-switched audio (for example Chinese and English in one recording) is transcribed and word-aligned in one pass, which the per-language Wav2Vec2 alignment stage cannot do.
• hotwords and initial_prompt become free-text context biasing, which corrects domain terms more reliably than token-level hotwords.

Why not:

• Roughly 10x slower than faster-whisper (about 12x realtime vs 116x on an RTX 3090). The full pipeline uses about 6 GB VRAM with the default 1.7B model (vs 4.6 GB for whisper large-v3), or about 3.6 GB with QWEN3_ASR_MODEL=Qwen/Qwen3-ASR-0.6B-hf.
• The requested Whisper model name is ignored and task=translate falls back to the whisper backend.
• Audio is processed in 90-second chunks (QWEN3_CHUNK_SECONDS); words spanning a chunk boundary may be split.

Weights (about 4.6 GB total) download to the cache volume on first use, the same as every other model. Nothing downloads unless the backend is enabled.

ASR_BACKEND=qwen3
# Optional overrides (defaults shown):
# QWEN3_ASR_MODEL=Qwen/Qwen3-ASR-1.7B-hf
# QWEN3_ALIGNER_MODEL=Qwen/Qwen3-ForcedAligner-0.6B-hf
# QWEN3_CHUNK_SECONDS=90
# QWEN3_DEFAULT_CONTEXT=          # standing vocabulary/context biasing text

Code-switched audio requires a per-request language. Passing any one of the mixed languages makes the backend force verbatim transcription (the other languages are still transcribed in their own script). Without a language, the model picks the dominant language per chunk and translates the rest into it; context hints do not prevent this. Speaker segments are always rebuilt at speaker-change boundaries on this backend (see RESEGMENT_BY_SPEAKER below for the whisper backend).

External ASR Backend (Experimental)

Setting ASR_BACKEND=external outsources only the transcription stage to an OpenAI-compatible API; alignment, diarization, speaker embeddings, and voice profiles keep running locally. Be aware that your audio is uploaded to the configured provider, which is why this is strictly opt-in.

ASR_BACKEND=external
EXTERNAL_ASR_BASE_URL=https://api.openai.com/v1
EXTERNAL_ASR_API_KEY=sk-...
EXTERNAL_ASR_MODEL=whisper-1
# EXTERNAL_ASR_MODE=transcriptions   # default; or "chat" for audio-input
                                     # chat models (OpenRouter, vLLM)

How word timestamps are produced depends on the provider response. Providers that return timestamped segments (whisper-1 verbose_json, Groq, self-hosted Whisper servers) feed the existing Wav2Vec2 alignment stage directly. Text-only providers (gpt-4o-transcribe, Voxtral or other audio models through OpenRouter's chat API, VibeVoice via vLLM) are word-timestamped by the Qwen forced aligner instead, which downloads on first use (~1.2 GB) and adds about 1 GB of VRAM. task=translate falls back to the whisper backend, and the requested Whisper model name is ignored.

Set EXTERNAL_ASR_ALIGNER=qwen to force the Qwen forced aligner even for timestamped responses; it is language-agnostic across its supported set and handles code-switched audio, unlike the per-language Wav2Vec2 models. The reverse is not configurable: Wav2Vec2 requires segment timestamps, so text-only responses always use the Qwen aligner.

Running the Service

# Start in simple mode (default)
docker compose up -d

# Or start in Ray Serve mode
# Set SERVE_MODE=ray in your .env file, then:
docker compose up -d

# View logs
docker compose logs -f

Note: When using Ray Serve mode, Docker Compose is configured with shm_size: 8g for Ray's shared memory object store. The Ray Dashboard is exposed on port 8265.

Monitoring and Logs

View Logs

# Real-time logs
docker compose logs -f

# Last 100 lines
docker compose logs --tail=100

# Specific container logs
docker logs whisperx-asr-api

Health Check

# Check service health
curl http://localhost:9000/health

# Response:
{
  "status": "healthy",
  "device": "cuda",
  "loaded_models": ["large-v3"],
  "serve_mode": "ray"
}

Prometheus Metrics

GET /metrics returns OpenMetrics text format suitable for a Prometheus scrape config:

curl http://localhost:9000/metrics
# # HELP whisperx_requests_total Total HTTP requests by endpoint and status
# # TYPE whisperx_requests_total counter
# whisperx_requests_total{endpoint="/asr",status="ok"} 12.0
# whisperx_request_duration_seconds_bucket{endpoint="/asr",le="60.0"} 11.0
# ...

Ray Serve caveat: /metrics is served by the ingress process. Whisper models load inside replica processes, so whisperx_loaded_models, whisperx_vram_allocated_bytes, and whisperx_model_evictions_total will read 0 (or be missing) in Ray Serve mode. These counters and gauges are per-process; without prometheus_client multi-process mode the ingress' registry never sees the replica's events. HTTP-level metrics (request counts, durations, audio sizes) are accurate in both modes because they are recorded inside the ingress handler. Use the Ray Dashboard at port 8265 for per-replica state, and tail serve/replica_* log files for eviction events. Multi-process metrics support is a planned follow-up.

In simple mode (SERVE_MODE=simple) all metrics including the eviction counter and loaded-model gauge work as expected, since the model cache and the /metrics handler share a process. Each Whisper model load pre-registers a whisperx_model_evictions_total{model="<name>"} 0 row so dashboards can graph the metric from the first load onward, not only after the first eviction.

The legacy JSON shape (queue/loaded model state) is still available at GET /queue-metrics for callers that depended on it.

Performance Monitoring

Monitor GPU usage:

# NVIDIA GPU stats
nvidia-smi -l 1

# Docker container stats
docker stats whisperx-asr-api

# Ray Dashboard (Ray mode only)
# Open http://localhost:8265 in your browser

Offline Use

This service can run completely offline after an initial setup with internet access. This is useful for air-gapped environments or when you want to avoid network latency.

Initial Setup (requires internet)

1. Start the container with internet access
2. Run at least one transcription request with diarization enabled to cache all models:

   curl -X POST http://localhost:9000/asr \
     -F "audio_file=@test.mp3" \
     -F "diarize=true"
   

3. This downloads and caches:
   • Whisper model (e.g., large-v3)
   • Alignment model (wav2vec2)
   • Pyannote speaker diarization models

Enable Offline Mode

Add HF_HUB_OFFLINE=1 to your docker-compose.yml environment section:

environment:
  - HF_HUB_OFFLINE=1
  # ... other environment variables

Important: This must be set directly in docker-compose.yml, not in the .env file.

Then restart the container:

docker compose down && docker compose up -d

The service will now operate without any network requests to Hugging Face.

Air-Gapped Transfer (no internet on the target machine)

If the machine that runs the service has no internet at all, populate the cache on an internet-connected machine and move it across as data. Do not rely on docker commit: the cache lives in a named Docker volume (whisperx-cache:/.cache), and docker commit captures only the container filesystem, not volume contents, so a committed-and-exported image arrives with an empty cache and fails to find the models at startup (the Swagger UI will also fail to load).

1. On the online machine, run one transcription with diarization (see above) so every model is cached.
2. Export the cache volume to a tarball:

   docker run --rm -v whisperx-cache:/cache -v "$PWD":/out alpine \
     tar czf /out/whisperx-cache.tgz -C /cache .
   

3. Transfer whisperx-cache.tgz and the image (docker save IMAGE | gzip > image.tgz) to the target machine.
4. On the target machine, load the image, extract the cache, and bind-mount it instead of a named volume:

   mkdir -p /srv/whisperx-cache
   tar xzf whisperx-cache.tgz -C /srv/whisperx-cache
   docker run -d --gpus all -p 9000:9000 \
     -e HF_HUB_OFFLINE=1 \
     -v /srv/whisperx-cache:/.cache \
     learnedmachine/whisperx-asr-service:latest
   

   On RTX 50xx / Blackwell cards use the :0.3.2-blackwell image and keep COMPUTE_TYPE=float16 (the CTranslate2 INT8 path is not supported on Blackwell).

What Gets Cached

Troubleshooting Offline Mode

If you see errors like Failed to resolve 'huggingface.co':

1. Ensure you ran a full transcription with diarization while online
2. Verify HF_HUB_OFFLINE=1 is set in docker-compose.yml (not .env)
3. Check the cache volume contains the models: docker exec whisperx-asr-api ls -la /.cache/ (faster-whisper models) and docker exec whisperx-asr-api ls -la /.cache/hub/ (pyannote/HuggingFace models)

Troubleshooting

GPU Not Detected

Symptom: Service runs on CPU despite having GPU

Solution:

# Verify NVIDIA Docker runtime
docker run --rm --gpus all nvidia/cuda:12.1.1-base-ubuntu22.04 nvidia-smi

# If fails, install nvidia-container-toolkit
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

Out of Memory Errors

Symptom: CUDA out of memory errors or VRAM exhaustion with large files

Solutions:

1. Reduce file size limit in .env: MAX_FILE_SIZE_MB=500 (default is 1000MB)
2. Use smaller model: small or medium instead of large-v3
3. Reduce batch size in .env: BATCH_SIZE=8 or BATCH_SIZE=4
4. Use int8 precision: COMPUTE_TYPE=int8 (lower quality but less memory)
5. Split large files: Process audio in smaller chunks before uploading
6. Disable diarization: For very large files, skip speaker diarization

Note: The service automatically clears GPU cache between operations to minimize VRAM buildup, but very large files (>500MB) can still cause issues.

Speaker Diarization Not Working

Symptom: No speaker labels in output

Solutions:

1. Verify HF_TOKEN is set correctly
2. Accept model user agreements on Hugging Face
3. Check logs for diarization errors: docker compose logs
4. Ensure diarize=true in request

Slow Processing

Symptom: Transcription takes too long

Solutions:

1. Use GPU instead of CPU (DEVICE=cuda)
2. Use smaller model for faster processing
3. Increase BATCH_SIZE (if you have VRAM)
4. Disable diarization if not needed: diarize=false

PyTorch 2.6 Weights Loading Error

Symptom: Error message containing Weights only load failed or GLOBAL omegaconf.listconfig.ListConfig was not an allowed global

This occurs due to a security change in PyTorch 2.6 where weights_only=True became the default for torch.load().

Solution:

Add this environment variable to your docker-compose.yml:

environment:
  - TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=true

Important: Setting this in .env file alone may not work - it must be set directly in docker-compose.yml under the environment section.

See WhisperX Issue #1304 for more details.

API Returns 500 Errors

Check logs:

docker compose logs whisperx-asr

Common causes:

• Invalid audio format (use ffmpeg to convert)
• Model not loaded (check VRAM, logs)
• Incorrect parameters (check API docs)

Supported Audio Formats

The service supports formats that WhisperX can process (via FFmpeg):

• Audio: MP3, WAV, M4A, FLAC, AAC, OGG, WMA
• Video: MP4, AVI, MOV, MKV, WebM (audio track extracted)
• Other: AMR, 3GP, 3GPP

Note: Large files (>1GB) may cause out-of-memory errors as files are loaded entirely into memory.

Security Notes

This service has NO built-in authentication or security features.

If exposing to a network:

• Use firewall rules to restrict access
• Consider putting behind a reverse proxy
• Store HF_TOKEN securely (use .env file, not hardcoded)

Maintenance

Updating WhisperX

If you run the prebuilt image, pull the new tag:

docker compose pull
docker compose up -d

If you build from source, the default build now installs PyTorch 2.7.1 from the cu126 wheel index (Pascal-Hopper). To target a different combo:

git pull
docker compose build --no-cache \
  --build-arg TORCH_VERSION=2.8.0 \
  --build-arg TORCH_INDEX_URL=https://download.pytorch.org/whl/cu128
docker compose up -d

Build args also work via Compose's build.args if you check that into your overlay file.

Clearing Cache

# Remove model cache
docker compose down -v
docker volume rm whisperx-asr-service_whisperx-cache

# Rebuild
docker compose up -d

Backup

Backup the cache volume to preserve downloaded models:

docker run --rm -v whisperx-asr-service_whisperx-cache:/cache \
  -v $(pwd):/backup ubuntu tar czf /backup/whisperx-cache-backup.tar.gz /cache

Stress Testing

A stress test script is included to measure throughput and latency under concurrent load:

# Default: 4 concurrent workers, all files in testfiles/
python tests/stress_test.py

# 8 concurrent workers, 3 rounds
python tests/stress_test.py --workers 8 --rounds 3

# Test OpenAI-compat endpoint
python tests/stress_test.py --endpoint openai

# Without diarization
python tests/stress_test.py --no-diarize

Place .mp3 files in the testfiles/ directory (gitignored). The report shows per-request latency, throughput in requests/minute, and the speedup from concurrent execution. See tests/README.md for full details.

License

This project is MIT licensed. See LICENSE for details.

WhisperX is licensed under BSD-4-Clause. See WhisperX repository for details.

Contributing

Contributions welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Submit a pull request

Support

For issues and questions:

• GitHub Issues: Create an issue
• WhisperX Issues: WhisperX repository

Credits

• WhisperX: m-bain/whisperX
• WhisperX Pyannote.audio 4 Support: sealambda/whisperX@feat/pyannote-audio-4 - This service uses sealambda's fork for pyannote.audio 4.0 compatibility
• OpenAI Whisper: openai/whisper
• Pyannote.audio: pyannote/pyannote-audio
• Docker WhisperX: jim60105/docker-whisperX

Changelog

v0.3.2 (2026-05-03)

Reported issues fixed

• Pascal/Blackwell image variants (#15): :latest ships PyTorch 2.7.1 (cu126, supports Pascal through Hopper). New :blackwell tag ships PyTorch 2.8.0 (cu128) for RTX 50xx. The Dockerfile now exposes TORCH_VERSION and TORCH_INDEX_URL build args and re-pins torch after the WhisperX install so the requested version sticks (the upstream fork was silently upgrading torch to 2.8 and breaking Pascal). CI publishes both variants per release with separate buildx cache scopes; either job can fail independently.
• Device-aware BATCH_SIZE default (#12): the default is now 16 on cuda and 2 on cpu. The hardcoded 16 was OOM-killing CPU runs (exit 137) on audio longer than ~30 minutes.
• Idle model eviction (#16): new MODEL_KEEP_ALIVE_SECONDS env var (default 0/disabled) unloads Whisper models that have been idle longer than the configured window. MODEL_EVICTION_INTERVAL_SECONDS controls sweep frequency (floor 30s). The next request that needs an evicted model reloads it transparently.
• Real Prometheus /metrics (#13): /metrics now returns OpenMetrics text instead of JSON. New histograms and counters cover request duration, status, audio duration/size, in-flight requests, loaded model count, model evictions, and VRAM. The previous JSON shape is preserved at /queue-metrics for callers that depended on it. See Prometheus Metrics for the full table and the Ray Serve caveat.

Other fixes shipped in v0.3.2

• /asr accepts the OpenAI-style aliases advertised by /v1/models (whisper-1, whisper-tiny, whisper-large-v3, ...). Previously these returned a 500 because the raw value was passed straight through to faster_whisper.WhisperModel.
• /v1/models is sourced from faster_whisper.available_models() instead of a hardcoded list, so the advertised set stays in sync with whatever engine version is installed (about 20 canonical names plus the whisper-1 alias).
• app.* log records from Ray Serve replicas now flow into the per-replica log file. Previously Loading WhisperX model: X, Starting transcription..., etc. were silently dropped because Ray Serve disables propagation on its own logger.
• whisperx_model_evictions_total{model="<name>"} is pre-registered with value 0 each time a model loads, so dashboards can graph the metric from the first load onward instead of only after the first eviction. Simple mode only; see the Ray Serve caveat.

New env vars

New build args

Tests

tests/test_v0_3_2.sh and tests/test_keep_alive.sh cover the new endpoints and eviction logic; both verified end-to-end against a Ray Serve container.

v0.3.0 (2026-02-28)

• Thread-safe model loading with double-checked locking for concurrent request safety
• Add Ray Serve mode for high-throughput ASR with cross-request batching
• Two pipeline strategies: replicate (full pipeline per GPU) and split (stage per GPU)
• Multi-GPU support via NUM_GPU_REPLICAS and per-stage replica/fraction config
• Refactor pipeline into shared stage functions (app/pipeline.py)
• Add async GPU queue with semaphore for simple mode (non-blocking event loop)
• Add /metrics endpoint for pipeline monitoring
• Add SERVE_MODE, PIPELINE_STRATEGY, NUM_GPU_REPLICAS env vars
• Add entrypoint.sh for automatic mode switching in Docker
• GPU pinning via NVIDIA_VISIBLE_DEVICES in local compose overrides
• Add stress test suite (tests/stress_test.py)

v0.2.0 (2025-01-21)

• Add /v1/models and /v1/audio/transcriptions endpoints for OpenAI API compatibility
• Add diarize parameter for broader API compatibility
• Add offline mode support and fix model caching
• Use Query() parameters to work as a drop-in replacement for other Whisper ASR services

v0.1.1alpha (2025-11-23)

• Initial release
• WhisperX integration with API wrapper
• Speaker diarization support
• Docker deployment