Cohere Transcribe ASR API

An HTTP API for transcribing audio files with the locally cached Cohere Transcribe models from HuggingFace:

The Cohere entries are 2B-parameter Conformer encoder-decoder models, served via transformers>=5.4.0 (AutoProcessor + CohereAsrForConditionalGeneration). The ivrit entry is a Whisper large-v3-turbo fine-tune for Hebrew in CTranslate2 format, served via a second backend, faster-whisper. Weights are loaded from the local HuggingFace cache (~/.cache/huggingface/hub/) — no downloads required (and enforced: ASR_LOCAL_FILES_ONLY=1 by default, so a cache miss fails loudly instead of fetching mid-request).

Security posture. The API itself implements no authentication. Access is gated in front of it, by a host firewall plus Cloudflare Zero Trust login; there is no unauthenticated public path to it. That is an accepted risk for this deployment, which is why the daemon still binds 0.0.0.0 by default. If you deploy it any other way, put an authenticating proxy in front of it, or start it with --host 127.0.0.1.

Endpoints

POST /v1/audio/transcriptions

Transcribes an uploaded audio file (OpenAI-style multipart form). Long-form audio (>35 s) is chunked and reassembled automatically.

Response:

{
  "text": " If not, there will be a big crisis between you and the European Parliament.",
  "model": "cohere-transcribe-03-2026",
  "language": "en",
  "duration_s": 5.44
}

Errors: 400 for unknown model, unsupported language, or undecodable audio; 500 for inference failures.

Backend notes: punctuation and max_new_tokens only apply to the Cohere models — the faster-whisper backend (ivrit-whisper-large-v3-turbo-ct2) ignores them (Whisper always punctuates, and long-audio windowing is handled internally instead of the ASR_CHUNK_BATCH mini-batch loop).

Long recordings and proxy timeouts (Cloudflare's 100 s): synchronous requests only work for audio short enough to finish within the proxy limit. For anything longer, submit with background=true — the request returns in under a second — and poll the job endpoint:

# 1. Submit (returns immediately)
curl -X POST https://asr.ninefive.cc/v1/audio/transcriptions \
  -H "Cookie: CF_Authorization=$CF_JWT" \
  -F "file=@long_recording.mp3" -F "background=true" -F "language=ar"
# → {"job_id": "070c7b86…", "status": "queued"}

# 2. Poll until "done" (each poll is a fast request, no proxy timeout)
curl -H "Cookie: CF_Authorization=$CF_JWT" \
  https://asr.ninefive.cc/v1/audio/transcriptions/jobs/070c7b86…
# → {"status": "queued"|"running"|"done"|"failed", "result": {...}|"error": "..."}

Verified through Cloudflare with a 24-minute mp3 (42 chunks): submit 0.8 s, job completes in the background, VRAM freed afterwards. Long-form generation runs in mini-batches of ASR_CHUNK_BATCH chunks (default 4) so activation memory stays bounded regardless of recording length.

Jobs are kept in process memory (lost on restart); one worker thread processes them in FIFO order, serialized with synchronous requests.

GET /v1/audio/transcriptions/jobs/{job_id}

Returns {"job_id", "status", ...} — result when done, error when failed.

GET /v1/models

Lists registered models, default languages, and whether each is loaded.

GET /health

Returns {"status": "ok"}.

Usage

# English (default model)
curl -X POST http://localhost:8000/v1/audio/transcriptions \
  -F "[email protected]" \
  -F "language=en"

# Arabic model
curl -X POST http://localhost:8000/v1/audio/transcriptions \
  -F "[email protected]" \
  -F "model=cohere-transcribe-arabic-07-2026" \
  -F "language=ar"

# No punctuation
curl -X POST http://localhost:8000/v1/audio/transcriptions \
  -F "[email protected]" -F "language=en" -F "punctuation=false"

Interactive API docs (FastAPI/Swagger): http://localhost:8000/docs

Web UI (/)

A user-facing page is served at / (API docs stay at /docs):

• Light/dark theme toggle (natural sand/olive palette; persisted in localStorage, follows prefers-color-scheme by default)
• Audio file upload (accept: wav, mp3, m4a, ogg, flac, webm, aac) and in-browser microphone recording (MediaRecorder → webm/ogg; the server converts via an ffmpeg fallback when librosa can't decode the container)
• Model picker with the -int8enc quantized variants preselected/recommended; language picker scoped to the chosen model family with full names (en - English, ar+en - Arabic / English (mixed, code-switched), …)
• Advanced settings (punctuation, max_new_tokens) tucked behind a collapsed details section
• Submits with background=true and polls, so even long recordings never hit proxy timeouts; the transcript appears in a formatted card (RTL-aware) with a Download .txt button

LLM post-cleanup (optional)

Pass cleanup=true (form field) to post-process the transcript with a local ollama LLM (gemma4:e2b at http://10.200.100.5:11434, override with OLLAMA_HOST / OLLAMA_MODEL). The prompts in llm-post-processing-system-prompt.txt / llm-post-processing-user-prompt.txt make it add punctuation/capitalization, format numbers/dates, and conservatively remove ASR artifacts while preserving every spoken word (including fillers), for English, Arabic, and code-switched text.

• Runs after the ASR model is evicted from VRAM, and the LLM is always unloaded afterwards (keep_alive: 0) — verified via /api/ps → empty and VRAM back at baseline.
• The result gains a cleanup object: cleaned_text, changes[] (type, original, replacement, reason, confidence), warnings[]. On LLM failure the transcription still succeeds and cleanup.error is set instead.
• Optional translation: set translate_lang (e.g. French) and optionally translate_modifier (e.g. Levantine, combined as "Levantine Arabic") — the core cleaning rules still run first, then the cleaned text is translated; cleaned_text holds the translation and cleanup.translation.target echoes the target. The web UI gates this behind a checkbox with a type-to-narrow language autocomplete (~95 major languages) and a 40-char-capped modifier box.
• Web UI: "Clean up transcript with LLM" checkbox → second result card with the cleaned text, per-change highlighted deltas (badge + strikethrough → replacement + reason + confidence), warnings, Copy and Download buttons.
• CLI: ./transcribe file.mp3 --cleanup also saves <name>.cleaned.txt; add --translate French [--translate-modifier ...] for translation.

Text to speech (Kokoro-82M)

The cached hexgrad/Kokoro-82M model is integrated for TTS, standalone or combined with transcription. It runs on CPU (fast at 82M params; the GPU stays free for ASR).

• POST /v1/audio/speech — form fields text, voice (default af_heart), speed (0.5–2.0) → audio/wav (24 kHz).
• GET /v1/tts/voices — all 54 cached voices with gender/language labels.
• Combined: pass speak_voice=<id> (and optional speak_speed) to /v1/audio/transcriptions — the result gains tts.audio_wav_b64, spoken from the cleaned/translated text when cleanup ran, else the raw transcript.
• Web UI: a "Text to speech" card (textarea, voice picker grouped by language, speed slider, inline player + wav download) and 🔊 Speak buttons on both transcript cards.
• CLI: ./transcribe file.mp3 --speak [voice] saves <name>.<voice>.wav.

Language coverage: Kokoro has voices for English (US/GB), Spanish, French, Hindi, Italian, Japanese, Mandarin, and Portuguese (BR) — no Arabic. Voice id prefix picks the language; the UI groups voices accordingly. Note the kokoro/misaki packages pin Python <3.13 and their English G2P needs spacy, so on this host they are installed with --no-deps --ignore-requires-python and tts.py routes all languages through misaki's espeak-based G2P (English homograph handling is slightly weaker than the spacy path as a result).

SILMA TTS (Arabic + English, voice cloning)

silma-ai/silma-tts (F5-TTS based, 150M) covers Arabic and English — filling Kokoro's Arabic gap. It clones voices from short reference clips instead of named voices: entries in voices/ (<id>.wav + <id>.txt transcript) appear as silma_* voices in /v1/tts/voices and the UI picker (group "Arabic + English (voice clone)"). Currently bundled: silma_ar_1, silma_ar_2 (Arabic), silma_en_1 (English). Add your own by dropping a clean 3–10 s reference wav + exact transcript into voices/ (mind the model's voice-consent license terms). SILMA runs on CPU (SILMA_TTS_DEVICE to override), auto-applies Arabic tashkeel via CATT, and is usable exactly like a Kokoro voice — standalone /v1/audio/speech calls or speak_voice=silma_* on transcriptions.

Host shims (see tts_silma.py): NeMo text normalizer is stubbed pass-through (pynini won't build on py3.14 — numbers/dates aren't normalized), and torchaudio.load is replaced with a soundfile reader (torchcodec's native lib won't load here).

Client script (./transcribe)

Wraps the whole flow — file selection, upload, background polling (immune to proxy timeouts), printing, and saving. Zero dependencies (Python standard library only, any Python 3.8+): copy this one file to any machine and run it, no venv or pip install needed. A GUI file picker (zenity) is used when no path is given and available, otherwise a terminal prompt.

./transcribe sermon.mp3                  # GUI file picker if no path given (zenity)
./transcribe sermon.mp3 --language ar    # saves sermon.cohere-transcribe-arabic-07-2026-int8enc.txt
./transcribe sermon.mp3 --output out.txt --no-punctuation
./transcribe sermon.mp3 --server http://127.0.0.1:8000   # local server, no auth

• Defaults to the public endpoint https://asr.ninefive.cc (override with --server or ASR_SERVER; also ASR_MODEL, ASR_LANGUAGE).
• For the Cloudflare-Access endpoint it needs the CF_Authorization JWT via --cookie, the CF_AUTHORIZATION env var, or a ./cf.jwt file (note: the cookie expires ~24 h after browser login). Localhost servers skip auth.
• Always submits with background=true and polls, so even hour-long files never hit the 100 s Cloudflare timeout. Progress dots while waiting; Ctrl+C aborts the client without killing the server-side job.
• The transcript is printed and saved next to the audio as <name>.<model-id>.txt unless --output is given.

Running

The asr-server script manages a daemonized (nohup + pidfile) server in userland — no systemd needed:

cd /home/nolan/cohere-asr
./asr-server start --device cuda --gpu 0 \
  --default-model cohere-transcribe-arabic-07-2026-int8enc
./asr-server status    # pid + /health + VRAM in use
./asr-server logs      # follow the server log (asr-server.log)
./asr-server stop      # SIGTERM, then SIGKILL if needed
./asr-server restart --device cpu --dtype float32 --port 8080

Plain uvicorn app:app --host 0.0.0.0 --port 8000 still works for foreground/debug runs.

Every request (or background job) loads the model it needs, transcribes, and then completely evicts the model from VRAM before finishing (verified: VRAM returns to the ~85 MiB CUDA-context baseline after every request, including error paths). This keeps the GPU free for co-located processes (e.g. ollama) and makes stale-model eviction bugs impossible — two models are never in VRAM at once. The trade-off: each request pays a model load (~10–40 s depending on page cache; INT8 variants also re-quantize). The whole load → generate → unload lifecycle runs under one lock, so requests are serialized and a model is never freed mid-generation.

Quantization (INT8 encoder / FP16 decoder)

The -int8enc variants reproduce the FluidInference/cohere-transcribe-03-2026-coreml hybrid scheme natively in PyTorch: per-output-channel symmetric INT8 encoder weights with FP16 activations; the decoder, embeddings, LayerNorms, and conv layers stay FP16 (precision-sensitive and small). Implemented in quant.py (Int8Linear), applied to every nn.Linear of the encoder at load time — no extra dependencies and no calibration data needed.

Why not serve the CoreML artifacts directly? That repo contains Apple CoreML packages (.mlpackage/.mlmodelc), and the CoreML runtime only exists on macOS/iOS — it cannot execute on this Linux host. The files remain in the HF cache for reference/Mac use; the API derives the same quantization from the original FP16 weights instead.

Measured on the Quadro P1000:

Both -int8enc variants produced transcripts identical to their FP16 counterparts on the demo files (English and Arabic). Encoder weight memory drops ~2× overall (INT8 vs FP16 on the encoder's linear layers).

Setup (already done in .venv/)

python3 -m venv .venv
.venv/bin/pip install --upgrade pip
# Torch cu126 build — required for the Quadro P1000, see notes
# torch / torchaudio first, from the PyTorch index — see requirements-torch.txt
.venv/bin/pip install torch==2.13.0+cu126 --index-url https://download.pytorch.org/whl/cu126
.venv/bin/pip install torchaudio==2.11.0+cpu --index-url https://download.pytorch.org/whl/cpu
.venv/bin/pip install -r requirements.txt

Hardware/software notes

• GPU: Quadro P1000 (4 GB, Pascal / sm_61). The stock PyTorch cu130 wheel ships no kernels for sm_61 — the cu126 build is required. The 2B model in fp16 (~4 GB) just fits: device_map="auto" places what fits on the GPU (observed ~3.9 GB used) and offloads any overflow to CPU RAM. CPU inference also works (ASR_DEVICE=cpu, fp32, needs ~8 GB RAM per model).
• Per-request VRAM release. Two 2B models (even INT8-quantized, ~2.6 GB each) plus inference temporaries do not fit in 4 GB together, and this GPU is shared with other processes (e.g. ollama) — so no model stays resident: VRAM is fully released after every request. One subtlety, if you touch this code: the unload must run in a frame that holds no references to the model, and not while an exception from inference is in flight (its traceback keeps the frame — and the model — alive). That is why transcribe is a thin wrapper around _transcribe that re-raises captured errors after unloading.
• Triton native kernels: enabled. Torch 2.13 routes some eager ops (e.g. bmm) through its native Triton kernels, which need a C compiler and the Python dev headers at runtime — gcc and python3.14-dev are installed on this host, so this path works. If you deploy elsewhere without them, set TORCH_DISABLE_NATIVE_JIT=1 to fall back to standard aten kernels.
• Verified stack: Python 3.14.4, torch 2.13.0+cu126, transformers 5.14.1, accelerate, fastapi, uvicorn, librosa/soundfile.

Verified results

• English demo (VoxPopuli, 5.4 s) → " If not, there will be a big crisis between you and the European Parliament." (identical for FP16 and -int8enc)
• Arabic demo (dialectal, code-switched, 3.6 s) → " ففي الحالة دي المسألة دي يعني more safe" (identical for FP16 and -int8enc)
• Error paths: unknown model → 400 with available ids; non-audio file → 400.
• Chunked vs single-batch parity (same arabic-07-2026-int8enc model, GPU mini-batches of 4 vs CPU single batch, 4 files incl. a 24-min/42-chunk one): 3 of 4 byte-identical; the 4th differed in exactly one word (الغروزني. vs الغروزنة. — an fp16-vs-fp32 rounding artifact, not chunking). Repro scripts and transcripts in comparison/.
• GPU vs CPU runtimes (same files, inference time excl. ~5–8 s model load): the P1000 is only ~1.1× faster than this host's CPU on short clips, and on the 24-min file the CPU's single big batch beats 11 sequential GPU mini-batches (303 s vs 314 s, both ≈ 4.7× real-time). Details in comparison/timing_summary.txt.
• --chunk-batch sweet spot: 4 (the default). Benchmarked 1–32 on the 24-min file (comparison/chunk_bench_summary.txt): inference time is flat (~316–328 s) from batch 1 through 8 — the workload is dominated by autoregressive decoder steps, so encoder batching buys nothing on the P1000 — while peak VRAM climbs 2.8 → 3.6 GB. Batch 4 is the fastest with comfortable headroom (3.1 GB peak); 16+ OOMs the 4 GB card. Drop to 1–2 only if ollama needs a few hundred extra MB during a transcription.

Background jobs

background=true returns 202 + job_id and the work happens on a single worker thread. The queue is bounded (ASR_MAX_QUEUED_JOBS, 503 when full), finished jobs expire after ASR_JOB_TTL_S, and DELETE /v1/audio/transcriptions/jobs/{id} drops one immediately — job results hold the transcript, so they are not kept indefinitely.

The worker's loop body is fully guarded: an unexpected error fails that one job and the thread keeps serving. If it ever did die, every job would sit in queued forever, which is why /health now reports worker_alive, queue_depth and seconds_since_last_job and returns degraded when the queue has not drained. Clients (transcribe, the web UI) also stop waiting after ASR_MAX_WAIT_S / 60 minutes instead of polling forever.

Data handling

• Transcripts are held in memory only, expire with their job, and are never written to the log — LLM failures log a length and a hash, not the payload.
• Logs are per-port files rotated by asr-server above 20 MiB, and contain client IPs, so treat them as personal data.
• voices/ (SILMA voice-clone references) is not tracked — the files are needed at runtime but are real voice samples, so keep them out of the repository and ship them out of band.
• comparison/*.txt are real transcripts kept as benchmark fixtures; they are a deliberate retention choice, not an oversight.

Tests

ASR_DEVICE=cpu ASR_DTYPE=float32 .venv/bin/python -m pytest -q

Covers input validation (upload caps, voice allowlist, token/language bounds), job lifecycle (TTL, DELETE, backpressure), worker resilience, health reporting, and the CLI's credential rules. No model is loaded and no network call is made.

Limitations (from the model card)

• Best with a single pre-specified language; no automatic language detection.
• No timestamps, no speaker diarization.
• Eager to transcribe non-speech noise — a VAD/noise gate upstream helps on quiet audio.

Project layout

cohere-asr/
├── app.py             # FastAPI service (single file)
├── static/index.html  # web UI served at /
├── quant.py           # INT8 encoder quantization (Int8Linear)
├── tts.py             # Kokoro-82M TTS (espeak G2P routing)
├── tts_silma.py       # SILMA TTS (Arabic/English voice cloning)
├── voices/            # SILMA reference clips (<id>.wav + <id>.txt) — NOT tracked
├── asr-server         # daemon manager (start/stop/status/logs)
├── transcribe         # CLI client (upload + poll + save)
├── tests/             # pytest suite (no model loads, no network)
├── .github/workflows/ # CI
├── comparison/        # benchmark scripts and transcripts
├── requirements.txt   # Python dependencies (pinned)
├── requirements-torch.txt  # torch/torchaudio pins (PyTorch index, install first)
├── .venv/             # virtualenv with the verified stack
├── asr-server.*.log   # daemon logs, per port (created at runtime)
├── asr-server.*.pid   # daemon pidfiles, per port (created at runtime)
└── README.md