ElBruno.Whisper

Run local Whisper speech-to-text in .NET 🎤

Transcribe audio to text in .NET using OpenAI's Whisper model. Powered by ONNX Runtime with automatic model download from HuggingFace.

Packages

Features

• 📦 Automatic model download — models are fetched from HuggingFace on first use
• 🔊 Multiple model sizes — tiny → base → small → medium → large (pick your speed/accuracy tradeoff)
• 🚀 Zero friction — works out of the box with sensible defaults (tiny.en)
• 🌍 Multilingual support — transcribe 99+ languages with multilingual models
• 💉 DI-friendly — register with AddWhisper() in ASP.NET Core
• 🧩 Microsoft.Extensions.AI ready — resolve ISpeechToTextClient for standard speech-to-text integration
• 🧵 Concurrent transcription — share one client with configurable queueing and pooled inference sessions
• 🔄 Incremental updates — get rolling provisional and committed text through GetStreamingTextAsync()
• 📼 Raw audio inputs — transcribe WAV streams, PCM16 byte streams, and Float32 memory without temporary files
• 📊 Progress reporting — track model downloads with real-time callbacks
• 🎯 English-optimized models — dedicated .en variants for best accuracy on English audio
• ⏱️ Timestamp-aware results — opt into segment and word timings for subtitle and caption workflows

Installation

dotnet add package ElBruno.Whisper

Quick Start

using ElBruno.Whisper;

// Create client (downloads tiny.en model on first run)
using var client = await WhisperClient.CreateAsync();

var result = await client.TranscribeAsync("audio.wav");
Console.WriteLine(result.Text);

First Run

The first time you create a WhisperClient, the model is downloaded from HuggingFace to your local cache directory (~75 MB - 3 GB depending on model size). This typically takes 10-60 seconds depending on your internet connection and chosen model.

Track download progress:

using var client = await WhisperClient.CreateAsync(
    progress: new Progress<ElBruno.HuggingFace.DownloadProgress>(p =>
    {
        if (p.Stage == ElBruno.HuggingFace.DownloadStage.Downloading)
            Console.WriteLine($"{p.CurrentFile}: {p.PercentComplete:F0}%");
        else
            Console.WriteLine($"{p.Stage}: {p.Message}");
    })
);

Subsequent runs load instantly from cache (%LOCALAPPDATA%/ElBruno/Whisper/models).

Model Selection

Whisper offers various model sizes. English-optimized models (.en suffix) are smaller and faster for English audio:

using var client = await WhisperClient.CreateAsync(new WhisperOptions
{
    Model = KnownWhisperModels.WhisperSmallEn
});

var result = await client.TranscribeAsync("english-audio.wav");
Console.WriteLine(result.Text);

Available Models

Use English-optimized (.en) models for:

• English audio only (slightly smaller, faster, better accuracy on English)

Use Multilingual models for:

• Non-English audio
• Mixed-language content
• Language auto-detection

Progress Tracking

Monitor both file downloads and transcription progress:

var downloadProgress = new Progress<ElBruno.HuggingFace.DownloadProgress>(p =>
{
    if (p.Stage == ElBruno.HuggingFace.DownloadStage.Downloading)
        Console.Write($"\r⬇️ {p.PercentComplete:F0}%");
    else
        Console.WriteLine($"\n✓ {p.Message}");
});

using var client = await WhisperClient.CreateAsync(progress: downloadProgress);

var result = await client.TranscribeAsync("audio.wav");
Console.WriteLine($"✓ Transcribed: {result.Text}");

Dependency Injection

Register Whisper options in ASP.NET Core or other DI-enabled applications:

builder.Services.AddWhisper(options =>
{
    options.Model = KnownWhisperModels.WhisperBaseEn;
    options.Concurrency.MaximumConcurrentRequests = 2;
});

AddWhisper() registers WhisperOptions, WhisperSpeechToTextClient, and ISpeechToTextClient. Create and share a WhisperClient yourself when you want direct control over startup and disposal.

Microsoft.Extensions.AI

Use the adapter when you want the standard ISpeechToTextClient contract:

using ElBruno.Whisper;
using Microsoft.Extensions.AI;

builder.Services.AddWhisper(options =>
{
    options.Model = KnownWhisperModels.WhisperTinyEn;
    options.Language = "en";
});

var speechToText = builder.Services
    .BuildServiceProvider()
    .GetRequiredService<ISpeechToTextClient>();

await using var audioStream = File.OpenRead("audio.wav");
var response = await speechToText.GetTextAsync(
    audioStream,
    new SpeechToTextOptions
    {
        SpeechLanguage = "en",
        AdditionalProperties = new()
        {
            ["elbruno.whisper.enable_timestamps"] = true
        }
    });

Console.WriteLine(response.Text);
Console.WriteLine(response.AdditionalProperties?["elbruno.whisper.detected_language"]);

The adapter keeps caller-owned streams open, supports cancellation, exposes SpeechToTextClientMetadata, and returns these response metadata keys through AdditionalProperties:

• elbruno.whisper.detected_language
• elbruno.whisper.audio_duration_ms
• elbruno.whisper.segments
• elbruno.whisper.words
• elbruno.whisper.model_id
• elbruno.whisper.execution_provider

Thread Safety and Concurrency

WhisperClient can now be shared across concurrent callers. By default it still processes one transcription at a time. Raise the concurrency limit to allow parallel work and reuse pooled ONNX sessions:

using var client = await WhisperClient.CreateAsync(new WhisperOptions
{
    Model = KnownWhisperModels.WhisperTinyEn,
    Concurrency = new WhisperConcurrencyOptions
    {
        MaximumConcurrentRequests = 2,
        QueueTimeout = TimeSpan.FromSeconds(15),
        EnableSessionPooling = true
    }
});

var results = await Task.WhenAll(
    client.TranscribeAsync("audio-1.wav"),
    client.TranscribeAsync("audio-2.wav"));

If all inference slots are busy longer than QueueTimeout, TranscribeAsync throws TimeoutException. Cancelling the request also aborts queue waiting and the next safe decode checkpoint.

Incremental Transcription

WhisperClient.GetStreamingTextAsync() runs Whisper over rolling windows and emits ordered updates with both stable and provisional text:

using var client = await WhisperClient.CreateAsync();

await foreach (var update in client.GetStreamingTextAsync(
    "audio.wav",
    new WhisperStreamingOptions
    {
        WindowSize = TimeSpan.FromSeconds(8),
        StepSize = TimeSpan.FromSeconds(1),
        ContextOverlap = TimeSpan.FromSeconds(2),
        UseLocalAgreement = true,
        AgreementIterations = 2
    }))
{
    Console.WriteLine($"Committed:   {update.CommittedText}");
    Console.WriteLine($"Provisional: {update.ProvisionalText}");

    if (update.IsFinal)
        Console.WriteLine("Final update received.");
}

Each update exposes:

• CommittedText — text that has stabilized across rolling windows
• ProvisionalText — the newest hypothesis that may still change
• Text — the combined transcript for the current update
• IsFinal — true exactly once, after the final flush

Limitations: Whisper is not a native streaming model. This API reads completed file or stream content, re-runs inference over overlapping windows, and uses local agreement to reduce duplicate committed text. Provisional text can still change between updates.

Realtime PCM and Memory Inputs

Use the explicit-audio overloads when your pipeline already has PCM in memory and you want to avoid temporary WAV files:

using var client = await WhisperClient.CreateAsync();

var pcm16Format = new WhisperAudioFormat(
    sampleRate: 48000,
    channels: 2,
    sampleFormat: WhisperAudioSampleFormat.Pcm16);

await using var rawAudioStream = File.OpenRead("call.raw");
var streamResult = await client.TranscribeAsync(rawAudioStream, pcm16Format);

ReadOnlyMemory<byte> pcmBytes = await File.ReadAllBytesAsync("call.raw");
var byteResult = await client.TranscribeAsync(pcmBytes, pcm16Format);

ReadOnlyMemory<float> monoFloatSamples = GetNormalizedSamples();
var floatResult = await client.TranscribeAsync(monoFloatSamples, sampleRate: 16000);

Notes:

• TranscribeAsync(Stream) auto-detects WAV headers and keeps the caller-owned stream open.
• Raw PCM byte streams require WhisperAudioFormat so the client can downmix and resample to Whisper's 16 kHz mono input.
• ReadOnlyMemory<float> overloads expect normalized PCM samples in the [-1, 1] range.

Blazor Components Package

Build speech-to-text interfaces quickly with ElBruno.Whisper.BlazorComponents:

dotnet add package ElBruno.Whisper.BlazorComponents

builder.Services.AddWhisper(options =>
{
    options.Model = KnownWhisperModels.WhisperBaseEn;
});
builder.Services.AddWhisperBlazorComponents();

Available components:

• WhisperModelSelector
• FileTranscriptionPanel
• LiveTranscriptViewer
• TranscriptionHistoryList

Transcription Result

The TranscriptionResult includes:

var result = await client.TranscribeAsync("audio.wav");

Console.WriteLine(result.Text);                    // Transcribed text
Console.WriteLine(result.DetectedLanguage);       // Detected language (for multilingual models)
Console.WriteLine(result.Duration);               // Audio duration

Enable timestamps to access both segment-level and word-level timing metadata:

using var client = await WhisperClient.CreateAsync(new WhisperOptions
{
    EnableTimestamps = true
});

var result = await client.TranscribeAsync("audio.wav");

foreach (var segment in result.Segments ?? [])
{
    Console.WriteLine($"[{segment.Start:mm\\:ss\\.ff} - {segment.End:mm\\:ss\\.ff}] {segment.Text}");

    foreach (var word in segment.Words)
    {
        Console.WriteLine($"  {word.Start:mm\\:ss\\.ff} - {word.End:mm\\:ss\\.ff}: {word.Text}");
    }
}

Word timings are derived from the timestamped transcript spans produced by Whisper. If a model returns text without explicit spans, the library falls back to a single full-duration segment and derives word timings within that range.

Troubleshooting

Model download fails?

• Check your internet connection
• For private HuggingFace models, set the HF_TOKEN environment variable

Out of memory?

• Use a smaller model (tiny or base instead of medium/large)
• Transcribe shorter audio files in chunks

For detailed troubleshooting, see docs.

Samples

What's New

1. Blazor Components package — Added ElBruno.Whisper.BlazorComponents with model selector, file transcription panel, live transcript viewer, and transcription history list.
2. Real-time Blazor transcription sample — Added chunk-based live transcription UX in the BlazorWhisper sample.
3. Timestamp-aware transcription results — Added optional segment timestamps in TranscriptionResult.
4. Model download progress reporting — Added structured progress callbacks during model download.
5. OIDC trusted publishing — Added secure NuGet publishing with GitHub Actions and NuGet/login.

Repository rule: keep last 5 important features

• The What’s New section must always list the latest 5 important repository/library features.
• Before every NuGet release, validate whether the release introduces a notable change and add/update an entry when needed.

Documentation

• Getting Started — installation, first steps, configuration
• API Reference — full API documentation
• Architecture — design decisions and internal structure
• Testing Guide — running tests, test organization, CI/CD pipeline
• Blazor Components — component API and integration guide
• Blazor Components Sample — walkthrough for the sample page
• Test Audio Files — audio resources for testing and transcription validation
• Image Prompts — prompts for generating blog and social media images
• Publishing — NuGet package publishing with OIDC

Building from Source

git clone https://github.com/elbruno/ElBruno.Whisper
cd ElBruno.Whisper
dotnet build ElBruno.Whisper.slnx
dotnet test ElBruno.Whisper.slnx --filter "Category!=Integration"

Testing

The repository includes comprehensive unit and integration tests:

Quick test run (unit tests, no model download):

dotnet test ElBruno.Whisper.slnx --filter "Category!=Integration"

Full test run (includes integration with real models):

dotnet test ElBruno.Whisper.slnx

Test audio files are provided in testdata/audio/ for validation and transcription testing. For details, see the Testing Guide.

🤝 Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📄 License

This project is licensed under the MIT License — see the LICENSE file for details.

Related Projects

• ElBruno.LocalLLMs — Run local LLMs in .NET
• ElBruno.HuggingFace — HuggingFace model utilities for .NET

🙏 Acknowledgments

• ONNX Runtime — inference engine
• OpenAI Whisper — speech-to-text model
• Hugging Face — model hosting and community
• ONNX Community — ONNX model conversions

👋 About the Author

Hi! I'm ElBruno 🧡, a passionate developer and content creator exploring AI, .NET, and modern development practices.

Made with ❤️ by ElBruno

If you like this project, consider following my work across platforms:

• 📻 Podcast: No Tienen Nombre — Spanish-language episodes on AI, development, and tech culture
• 💻 Blog: ElBruno.com — Deep dives on embeddings, RAG, .NET, and local AI
• 📺 YouTube: youtube.com/elbruno — Demos, tutorials, and live coding
• 🔗 LinkedIn: @elbruno — Professional updates and insights
• 𝕏 Twitter: @elbruno — Quick tips, releases, and tech news