Voice Agent With NVIDIA Open Models

This repo is sample code for building voice agents with three NVIDIA open source models:

• Nemotron Speech ASR
• Nemotron 3 Nano LLM
• Magpie TTS (Preview)

Run locally on an NVIDIA DGX Spark or RTX 5090. Or deploy to the cloud with Modal and Pipecat Cloud.

Accompanying blog posts:

• Nemotron Speech ASR Open Source Model Launch Post
• More About Voice Agent Architectures and This Agent's Design

Quick start - Run everything locally (DGX Spark or RTX 5090)

1. Build the Unified Container

docker build -f Dockerfile.unified -t nemotron-unified:cuda13 .

Build time: 2-3 hours (builds PyTorch, NeMo, vLLM, llama.cpp from source for CUDA 13.1 / Blackwell).

2. Start the Container

# Start with default Q8 model (auto-detected from HuggingFace cache)
./scripts/nemotron.sh start

# Or specify a model explicitly
./scripts/nemotron.sh start --model ~/.cache/huggingface/hub/models--unsloth--Nemotron-3-Nano-30B-A3B-GGUF/snapshots/.../Q8_0.gguf

# Start with vLLM instead of llama.cpp (requires ~72GB VRAM)
./scripts/nemotron.sh start --mode vllm

3. Run the Voice Bot

uv run pipecat_bots/bot_interleaved_streaming.py

Open http://localhost:7860/client in your browser.

Quick start - Deploy to Cloud with Modal and Pipecat Cloud

Modal (Services)

1. Prerequisites

Create a Modal account if you don't have one.

Then, install the necessary dependencies using uv with optional dependency group modal and authenticate your account.

# Install Modal and Pipecat Cloud dependencies
uv sync --extra modal --extra bot

# Authenticate with Modal
modal setup

2. Deploy Services to Modal

# Deploy ASR service
modal deploy -m src.nemotron_speech.modal.asr_server_modal

# Deploy TTS service
modal deploy -m src.nemotron_speech.modal.tts_server_modal

# Deploy vLLM service
modal deploy -m src.nemotron_speech.modal.vllm_modal

The ASR deployment takes about 30 seconds to cold-start, 60 seconds for TTS, and about 3 minutes for vLLM. You can uncomment the min_containers = 1 input to the Modal Function and Cls decorators to ensure that bots can start up quickly for production or development.

3. Run the bot locally or using Pipecat Cloud (see below)

uv run -m pipecat_bots.modal.bot_modal

Pipecat Cloud (Bot)

[!NOTE] Sign up for a Pipecat Cloud account here

1. Login to your Pipecat Cloud account using the CLI

# Install Pipecat Cloud package
uv sync --extra bot
# Or Pipecat Cloud and Modal
uv sync --extra bot --extra modal

# Login
pipecat cloud auth login

2. Create a new secret set with the necessary API keys

pipecat cloud secrets set gdx-spark-bot-secrets \
  NVIDIA_ASR_URL=wss:// \
  NVIDIA_LLM_URL=https:// \
  NVIDIA_TTS_URL=wss://

Alternatively, create your secret set from a .env file:

pipecat cloud secrets set gdx-spark-bot-secrets --file .env

3. Create image pull secret

Image pull secrets are used to authenticate with private Docker registries when deploying agents. See docs.

pipecat cloud secrets image-pull-secret gdx-spark-bot-pull-secret https://index.docker.io/v1/

Optional: Create a PCC deploy toml:

To speed up deployment you can create a pcc-deploy.toml in the project root. This file is read by the Pipecat CLI to pre-fill command arguments:

agent_name = "gdx-spark-bot"
image = "your-docker-repository/gdx-spark-bot:latest"
secret_set = "gdx-spark-bot-secrets"
image_credentials = "gdx-spark-bot-pull-secret"
agent_profile = "agent-1x"

[scaling]
	min_agents = 1

4. Build and push Docker image

docker build -f Dockerfile.bot -t gdx-spark-bot:latest .

# Optional: tag image
docker tag gdx-spark-bot:latest your-docker-repository/gdx-spark-bot:latest

# Push to image repository e.g. Docker Hub
docker push your-docker-repository/gdx-spark-bot:latest

5. Deploy

Run deploy command:

pipecat cloud deploy

# ...or if not using pcc-deploy.toml

pipecat cloud deploy gdx-spark-bot your-docker-repository/gdx-spark-bot:latest \
--credentials gdx-spark-bot-pull-secret \
--secrets gdx-spark-bot-secrets \
--profile agent-1x

6. Start bot using CLI

Create a public access key for Pipecat Cloud. Set this is a the default key when prompted:

pipecat cloud organizations keys create

Start an active session with your deployed bot:

pipecat cloud agent start gdx-spark-bot --use-daily

See docs for REST and Python usage.

Bot Variants

Three bot implementations are available:

Transport Options

All bots support multiple transport backends via the -t flag:

bot_interleaved_streaming.py / bot_simple_vad.py

bot_vllm.py

Pipecat Bot Components

Custom services in pipecat_bots/:

Local Container Management

Use ./scripts/nemotron.sh to manage the container:

# Start the container
./scripts/nemotron.sh start [OPTIONS]
  --mode MODE         LLM mode: llamacpp-q8 (default), llamacpp-q4, vllm
  --model PATH        Path to model file
  --no-asr            Disable ASR service
  --no-tts            Disable TTS service
  --no-llm            Disable LLM service
  -f, --foreground    Run in foreground (default: detached)

# Stop the container
./scripts/nemotron.sh stop

# Restart the container
./scripts/nemotron.sh restart [OPTIONS]

# Check status
./scripts/nemotron.sh status

# View logs
./scripts/nemotron.sh logs          # All logs interleaved
./scripts/nemotron.sh logs asr      # ASR logs only
./scripts/nemotron.sh logs tts      # TTS logs only
./scripts/nemotron.sh logs llm      # LLM logs only

# Open shell in container
./scripts/nemotron.sh shell

# Show help
./scripts/nemotron.sh help

Service Endpoints

Building the Container

# Build the unified container (2-3 hours)
docker build -f Dockerfile.unified -t nemotron-unified:cuda13 .

The build compiles from source for CUDA 13.1 / Blackwell (sm_121):

• PyTorch (with NVRTC support)
• torchaudio
• NeMo ASR/TTS
• vLLM
• llama.cpp

Model Requirements

Download LLM models (ASR and TTS are auto-downloaded on first run):

# GGUF quantized models (Q8 and Q4 variants for llama.cpp)
huggingface-cli download unsloth/Nemotron-3-Nano-30B-A3B-GGUF

# BF16 full precision (for vLLM)
huggingface-cli download nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16

Architecture

For detailed architecture documentation including frame flow, protocols, and timing diagrams, see docs/streaming-pipeline-architecture.md.

Troubleshooting

LLM crashes or stalls:

• The buffered LLM service uses single-slot operation (--parallel 1)
• Ensure adequate VRAM for context size (default 16384 tokens)
• Check for httpx connection issues if generation hangs

vLLM takes 10-15 minutes to start:

• This is normal for first startup (model loading, kernel compilation)
• Set SERVICE_TIMEOUT=900 if needed

vLLM DNS resolution issues:

• The container uses --network=host in vLLM mode to avoid DNS issues with HuggingFace