Talky Talky

A comprehensive audio MCP (Model Context Protocol) server for AI agents. Generate speech, transcribe audio, clone voices, analyze speech quality, design soundscapes, and manage audio assets—all through a standardized interface that works with any MCP-compatible client.

Quick Setup with AI Agents

Copy and paste this prompt to your AI agent (Claude Code, Cursor, Windsurf, etc.) to have it automatically configure Talky Talky:

Install and configure the Talky Talky MCP server for audio capabilities.

1. Clone the repo: git clone https://github.com/shawnrushefsky/talky-talky.git
2. Find the full path to uv: which uv (e.g., /Users/username/.local/bin/uv)
3. Add to my MCP configuration:

   For Claude Desktop (GUI app - MUST use full path to uv):
   {
     "mcpServers": {
       "talky-talky": {
         "command": "/full/path/to/uv",
         "args": ["run", "--directory", "<path-to-talky-talky>", "--extra", "macos-full", "talky-talky"]
       }
     }
   }

   For CLI tools (Claude Code, .mcp.json):
   {
     "mcpServers": {
       "talky-talky": {
         "command": "uv",
         "args": ["run", "--directory", "<path-to-talky-talky>", "--extra", "tts", "--extra", "transcription", "--extra", "analysis", "talky-talky"]
       }
     }
   }

4. Replace paths with actual values
5. Restart the application and verify by checking TTS availability

Platform extras:
- macOS: macos-full (TTS + transcription + analysis, excludes CUDA-only engines)
- Linux with CUDA: linux-cuda-full (all engines including CUDA-only)
- CPU only: cpu-full

Requirements: Python 3.11+, ffmpeg, GPU recommended for TTS/transcription engines.

---

Table of Contents

• Features
• Installation
• Configuration
• Available Tools
  • TTS Engine Tools
  • Speech Generation Tools
  • Song Generation Tools
  • Transcription Tools
  • Audio Analysis Tools
  • Audio Design Tools
  • Voice Modulation Tools
  • Audio Utility Tools
  • Audio Asset Management Tools
• TTS Engine Guide
• Transcription Engine Guide
• Usage Examples
• Development
• License

Features

Text-to-Speech (11 Engines)

Song Generation (2 Engines)

Speech-to-Text (2 Engines)

Audio Analysis

• Speech Quality: MOS prediction, noisiness, discontinuity, coloration
• Emotion Detection: 9 emotions (angry, happy, sad, surprised, etc.)
• Voice Similarity: Speaker verification and comparison
• TTS Verification: Automated quality checks for generated audio

SFX Analysis

• Loudness: Peak, RMS, LUFS, dynamic range, true peak
• Clipping Detection: Find digital distortion regions
• Spectral Analysis: Frequency content, brightness, energy distribution
• Silence Detection: Leading/trailing silence, gaps, content boundaries

Audio Design

• Mixing: Layer multiple tracks with volume control
• Effects: EQ (lowpass, highpass, bass, treble), reverb, echo, speed
• Fades: Fade in/out with configurable duration
• Overlays: Position-based audio layering
• Crossfades: Smooth transitions between segments
• Trimming: Auto-detect content boundaries, remove silence
• Silence Insertion: Add controlled pauses between segments

Voice Modulation

• Pitch Shifting: Change pitch without affecting speed (±12 semitones)
• Time Stretching: Change speed without affecting pitch
• Voice Effects: Robot, chorus, vibrato, flanger, telephone, and more
• Formant Shifting: Change voice character (masculine/feminine)

Note: Maya1 can achieve pitch, pacing, and timbre variations directly through voice descriptions (e.g., "high pitch, fast pacing, gravelly timbre"). Use the voice modulation tools for post-processing or with engines that don't have built-in voice control.

Audio Asset Management

• Local Library: Index and search local folders with SQLite FTS5
• Freesound.org: Search and download CC-licensed sounds
• Jamendo: 500k+ CC-licensed music tracks for free (non-commercial)
• License Tracking: CC0, CC-BY, CC-BY-NC, CC-BY-SA attribution
• Tagging: Manual and AI-powered auto-tagging

Audio Utilities

• Format Conversion: WAV, MP3, M4A with ffmpeg
• Concatenation: Join files with optional gaps
• Normalization: Broadcast standard (-16 LUFS)
• Playback: System default audio player

Cross-Platform Support

• CUDA: Full support with GPU acceleration
• MPS: Apple Silicon support for most engines
• CPU: Fallback for all engines (slower)

Installation

Prerequisites

• Python 3.11+ (required for TTS library compatibility)
• ffmpeg (required for audio processing)
• GPU (recommended for TTS and transcription)

Don't have Python 3.11+? Use uv which auto-manages Python versions:

curl -LsSf https://astral.sh/uv/install.sh | sh
uv run --extra tts talky-talky

Platform-Specific Installation

git clone https://github.com/shawnrushefsky/talky-talky.git
cd talky-talky

# macOS (Apple Silicon or Intel)
pip install -e ".[macos-full]"     # TTS + transcription + analysis

# Linux with NVIDIA CUDA GPU
pip install -e ".[linux-cuda-full]" # All engines including CUDA-only

# CPU only (no GPU)
pip install -e ".[cpu-full]"        # Excludes CUDA-only engines

Individual Components

# TTS engines
pip install -e ".[maya1]"           # Voice design
pip install -e ".[chatterbox]"      # Voice cloning (includes Turbo)
pip install -e ".[xtts]"            # Multilingual cloning
pip install -e ".[kokoro]"          # Pre-built voices
pip install -e ".[seamlessm4t]"     # Multilingual + translation
pip install -e ".[mira]"            # Fast cloning (CUDA only)
pip install -e ".[soprano]"         # Ultra-fast (CUDA only)

# Transcription
pip install -e ".[whisper]"         # OpenAI Whisper
pip install -e ".[faster-whisper]"  # 4x faster Whisper

# Analysis
pip install -e ".[emotion2vec]"     # Emotion detection
pip install -e ".[resemblyzer]"     # Voice similarity
pip install -e ".[nisqa]"           # Speech quality

# Song Generation (CUDA required for LeVo)
pip install -e ".[songgen]"         # LeVo + ACE-Step

# Combined
pip install -e ".[tts]"             # All TTS engines
pip install -e ".[transcription]"   # All transcription engines
pip install -e ".[analysis]"        # All analysis engines

Using uv (Recommended)

git clone https://github.com/shawnrushefsky/talky-talky.git
cd talky-talky

# Run directly without installing
uv run --extra macos-full talky-talky

Using Docker

# Basic image (audio utilities only)
docker pull ghcr.io/shawnrushefsky/talky-talky:latest
docker run -i ghcr.io/shawnrushefsky/talky-talky:latest

# With GPU access
docker run -i --gpus all talky-talky-cuda

Configuration

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS):

{
  "mcpServers": {
    "talky-talky": {
      "command": "/full/path/to/uv",
      "args": ["run", "--directory", "/path/to/talky-talky", "--extra", "macos-full", "talky-talky"]
    }
  }
}

macOS Note: GUI apps don't inherit shell PATH. Use full path to uv (which uv).

Claude Code / CLI

Add to .mcp.json or ~/.claude/settings.json:

{
  "mcpServers": {
    "talky-talky": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/talky-talky", "--extra", "tts", "--extra", "transcription", "talky-talky"]
    }
  }
}

Available Tools

TTS Engine Tools

Speech Generation Tools

Song Generation Tools

Transcription Tools

Audio Analysis Tools

Speech Analysis

SFX Analysis

Audio Design Tools

Voice Modulation Tools

Audio Utility Tools

Audio Asset Management Tools

TTS Engine Guide

Maya1 (Voice Design)

Create unique voices from natural language descriptions with 20+ emotion tags.

speak_maya1(
    text="I can't believe it! <laugh> We did it!",
    output_path="output.wav",
    voice_description="Excited young woman, American accent, energetic"
)

Emotion Tags: <laugh> <chuckle> <sigh> <gasp> <whisper> <angry> <yell> <cry> <cough>

Chatterbox (Voice Cloning)

Zero-shot voice cloning with emotion control. 23 languages.

speak_chatterbox(
    text="Hello! [chuckle] Nice to meet you.",
    output_path="output.wav",
    reference_audio_paths=["reference.wav"],
    exaggeration=0.6  # 0.0-1.0+ for expressiveness
)

SeamlessM4T v2 (Multilingual + Translation)

Generate speech in 35 languages with optional translation.

# Pure TTS
speak_seamlessm4t(
    text="Hello world",
    output_path="hello.wav",
    language="en"
)

# Translate English to French speech
speak_seamlessm4t(
    text="Hello world",
    output_path="bonjour.wav",
    src_language="en",
    language="fr",
    speaker_id=5  # 200 speaker options
)

Kokoro (Pre-built Voices)

54 high-quality voices, 8 languages, lightweight (82M).

speak_kokoro(
    text="Welcome!",
    output_path="output.wav",
    voice="af_heart",  # American female
    speed=1.0
)

Voice Format: [lang][gender]_[name] (e.g., af_heart, bm_george, jf_alpha)

See CLAUDE.md for complete documentation on all engines.

Transcription Engine Guide

Faster-Whisper (Recommended)

4x faster than standard Whisper with same accuracy.

transcribe_audio(
    audio_path="speech.wav",
    engine="faster_whisper",
    model_size="large-v3",  # tiny, base, small, medium, large-v3
    language="en"  # or None for auto-detection
)

With Timestamps

transcribe_with_timestamps(
    audio_path="speech.wav",
    word_level=True  # Get word-level timing
)

TTS Verification

verify_tts_output(
    audio_path="generated.wav",
    expected_text="Hello world",
    similarity_threshold=0.8
)

Usage Examples

Generate Narration with Custom Voice

Generate speech saying "Welcome to the future of AI" with a deep male narrator voice using Maya1

Clone a Voice

Clone the voice from sample.wav using Chatterbox and say "This is voice cloning" with high expressiveness

Transcribe Audio

Transcribe the audio file interview.wav with word-level timestamps

Create a Podcast Intro

1. Generate intro music search for "podcast intro" sounds
2. Generate speech: "Welcome to Tech Talk" with a professional voice
3. Mix the music and speech together with music at 20% volume
4. Add fade in/out effects

Analyze Speech Quality

Analyze the speech quality of generated.wav and check if it meets broadcast standards

Translate and Speak

Use SeamlessM4T to translate "Hello, how are you?" from English to Japanese speech

Voice Modulation

1. Generate speech with Maya1 saying "Hello there"
2. Create a robot version using the robot voice effect
3. Create a deeper version by shifting formants to 0.8
4. Create a chipmunk version by pitching up 6 semitones

Generate a Song

Generate a pop song with LeVo using these lyrics:
[intro-short] ; [verse] Walking through the city lights. Stars are shining bright tonight ;
[chorus] This is our moment. Nothing can stop us now ; [outro-short]

Use description: "female, pop, upbeat, synth"

Generate Music with ACE-Step (Apple Silicon)

Generate an instrumental jazz track with ACE-Step:
- Style prompt: "instrumental, jazz, piano, smooth, relaxing"
- Duration: 60 seconds

Development

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Lint and format
ruff check talky_talky
ruff format talky_talky

This project uses uv for package management.

License

MIT

Individual engines have their own licenses:

• Apache-2.0: Maya1, Kokoro, Soprano, CosyVoice3
• MIT: Chatterbox, MiraTTS, VibeVoice, LeVo, ACE-Step
• CPML: XTTS-v2 (Coqui Public Model License)
• CC-BY-NC-4.0: SeamlessM4T v2 (non-commercial only)

Contributing

Contributions welcome! Please open an issue or submit a pull request.