NVIDIA Voice Agent (C#)

A real-time voice agent built with ASP.NET Core 10 that performs Speech-to-Text (ASR), LLM processing, and Text-to-Speech (TTS) using NVIDIA NIM models via ONNX Runtime and TorchSharp.

Ported from: nvidia-transcribe/scenario5

Features

• Real-time Speech Recognition — NVIDIA Parakeet-TDT-0.6B-V2 via ONNX Runtime
• PersonaPlex LLM — NVIDIA PersonaPlex-7B-v1 full-duplex speech-to-speech AI (NEW ✨)
• Alternative LLMs — Phi-3-mini-4k / TinyLlama with 4-bit quantization
• Text-to-Speech — FastPitch + HiFiGAN voice synthesis
• WebSocket Streaming — Bi-directional audio via /ws/voice and /ws/logs
• Browser UI — Models panel with download status, progress bars, and disk paths
• Auto Model Download — Fetches models from HuggingFace on first run
• Mock Mode — Full development workflow without downloading models
• GPU Acceleration — CUDA with automatic CPU fallback
• Voice Personas — 18 pre-packaged voices with PersonaPlex
• Debug Mode — Record conversations to disk for testing and analysis (NEW ✨)
• Long-Form Audio Support — Automatic chunking for audio longer than 60 seconds (NEW ✨)

Long-Form Audio Support

The ASR model now supports arbitrarily long audio through intelligent overlapping chunk processing:

• ✅ Automatic chunking for audio > 60 seconds
• ✅ Intelligent overlap (2s) minimizes word-split artifacts
• ✅ Transparent merging preserves sentence boundaries & punctuation
• ✅ Zero code changes — works transparently

Example: Transcribe a 30-minute podcast in one call:

// Just pass any audio length — chunking happens automatically
var transcript = await asrService.TranscribeAsync(thirtyMinuteAudio);
// Result: Full transcription across multiple chunks, no duplicates

Performance: ~100-300 seconds for 30-minute real-world audio (varies by CPU/GPU).

📖 See Audio Chunking Guide for configuration, troubleshooting, and performance tips.

Prerequisites

• .NET 10.0 SDK
• (Optional) NVIDIA GPU with CUDA 11.8+ for GPU acceleration

Quick Start

# Clone and build
git clone https://github.com/elbruno/nvidia-voiceagent-cs.git
cd nvidia-voiceagent-cs
dotnet build

# (First-time) install Python helpers for model preparation
pip install -r scripts/onnx/requirements.txt

# (First-time) prepare Parakeet-TDT ASR model
python scripts/onnx/patch_encoder.py --model-dir model-cache/parakeet-tdt-0.6b/onnx
python scripts/onnx/patch_decoder.py --model-dir model-cache/parakeet-tdt-0.6b/onnx
python scripts/onnx/extract_vocab.py --model-dir model-cache/parakeet-tdt-0.6b

# (Windows) run all ASR prep steps at once
pwsh -File scripts/onnx/prepare-models.ps1 -ModelCachePath model-cache

# Run the app
cd NvidiaVoiceAgent
dotnet run

Open http://localhost:5003 in your browser. The app auto-downloads models into ModelHub:ModelCachePath on first run.

First-time setup: See the full step-by-step guide at docs/guides/model-preparation.md. No GPU? The app works on CPU and falls back to Mock Mode if models are unavailable.

Run Tests

dotnet test

Tests are configuration-driven. Core tests read tests/NvidiaVoiceAgent.Core.Tests/appsettings.Test.json (mirrors the main app) and skip real-model tests when models are not available.

Useful filters

dotnet test --filter "FullyQualifiedName~MockMode"
dotnet test --filter "FullyQualifiedName~RealModel"
dotnet test --filter "FullyQualifiedName~RealModelIntegrationTests"

See tests/NvidiaVoiceAgent.Core.Tests/README.md for the full test configuration guide.

Supported Models

PersonaPlex Model

PersonaPlex-7B-v1 is NVIDIA's state-of-the-art full-duplex speech-to-speech conversational AI:

• 7 billion parameters based on Moshi architecture
• Ultra-low latency: ~170ms time-to-first-token
• Voice control: 18 pre-packaged voice personas + custom voice cloning
• Persona prompting: Text prompts define conversation role/style
• Gated access: Requires accepting NVIDIA's license on HuggingFace

To use PersonaPlex:

1. Accept the license at https://huggingface.co/nvidia/personaplex-7b-v1

2. Generate a HuggingFace token with read access

3. Add token to appsettings.json:

   {
     "ModelHub": {
       "HuggingFaceToken": "hf_your_token_here"
     }
   }
   

4. Download via the UI or API: POST /api/models/PersonaPlex-7B-v1/download

📖 Detailed Setup Guide: See HuggingFace Token Setup Guide for complete instructions, troubleshooting, and security best practices.

Note: PersonaPlex currently runs in mock mode. TorchSharp integration for actual inference is planned.

Configuration

Edit NvidiaVoiceAgent/appsettings.json:

{
  "ModelHub": {
    "AutoDownload": true,           // Download models on startup
    "UseInt8Quantization": true,    // Prefer quantized models
    "ModelCachePath": "model-cache", // Local cache directory
    "HuggingFaceToken": null        // Required for gated models (PersonaPlex)
  },
  "ModelConfig": {
    "UseGpu": true,                 // CUDA acceleration
    "Use4BitQuantization": true,    // LLM quantization
    "PersonaPlexVoice": "voice_0"   // Default PersonaPlex voice (0-17)
  },
  "DebugMode": {
    "Enabled": false,               // Record conversations for testing
    "AudioLogPath": "logs/audio-debug",
    "SaveIncomingAudio": true,      // Save user voice
    "SaveOutgoingAudio": true,      // Save TTS responses
    "SaveMetadata": true,           // Save conversation metadata
    "MaxAgeInDays": 7               // Auto-delete old recordings
  }
}

Project Structure

nvidia-voiceagent-cs/
├── NvidiaVoiceAgent/              # ASP.NET Core Web App (UI, WebSockets, endpoints)
├── NvidiaVoiceAgent.Core/         # ML/Audio class library (ASR, AudioProcessor, MelSpectrogram)
├── NvidiaVoiceAgent.ModelHub/     # Model download library (HuggingFace integration)
├── tests/                         # xUnit test projects
└── docs/                          # Detailed documentation

Documentation

Contributing

1. Fork the repository
2. Create a feature branch: git checkout -b feature/my-feature
3. Commit changes and open a Pull Request

See the Developer Guide for coding standards.

License

MIT License — see LICENSE for details.

Credits

• Original: nvidia-transcribe (Python)
• Models: NVIDIA Parakeet, PersonaPlex, FastPitch, HiFiGAN
• Runtime: ONNX Runtime, TorchSharp, ASP.NET Core

Related Links

• PersonaPlex Research Page
• PersonaPlex on HuggingFace
• PersonaPlex GitHub