ZeroVoice: Real-time Speech-to-Speech Communication

ZeroVoice is a real-time speech-to-speech communication application built with modern web technologies. It allows users to interact with an AI voice assistant through natural language, featuring voice activity detection, speech-to-text transcription, large language model integration for responses, and text-to-speech synthesis.

Features

• Real-time Speech-to-Speech: Seamless bidirectional audio communication.
• Voice Activity Detection (VAD): Intelligently detects speech segments using Silero VAD.
• Automatic Speech Recognition (ASR): Transcribes user's speech to text using HuggingFace Transformers.js (Whisper base model).
• Large Language Model (LLM) Integration: Generates conversational responses using either:
  • Browser-based SmolLM2-1.7B-Instruct model (default)
  • Local LM Studio API for more powerful models
• Text-to-Speech (TTS): Synthesizes AI responses into natural-sounding speech using KokoroTTS.
• Responsive UI: Built with React and Tailwind CSS for a modern and adaptive user experience.
• Voice Selection: Users can choose from various available voices for the AI assistant.
• LLM Mode Toggle: Switch between browser-based and LM Studio models with a simple toggle.
• Call Timer: Tracks the duration of the conversation.

Technologies Used

• Frontend:
  • React: A JavaScript library for building user interfaces.
  • Tailwind CSS: A utility-first CSS framework for rapid UI development.
  • Vite: A fast frontend build tool.
• Speech Processing & AI:
  • @huggingface/transformers.js: For VAD (Silero VAD), ASR (Whisper), and LLM (SmolLM2-1.7B-Instruct).
  • KokoroTTS: For high-quality text-to-speech synthesis.
  • Web Audio API & AudioWorklets: For real-time audio processing in the browser.

Project Structure

• src/App.jsx: The main React component, handling UI, audio stream management, and state logic.
• src/worker.js: A Web Worker responsible for handling speech processing (VAD, ASR, LLM, TTS) to offload heavy computations from the main thread.
• src/constants.js: Defines various constants used across the application, especially for audio processing parameters.
• src/play-worklet.js: AudioWorklet for managing audio playback buffering.
• src/vad-processor.js: AudioWorklet for Voice Activity Detection processing.
• public/: Static assets like logo.png.

Setup and Installation

To get the project up and running on your local machine, follow these steps:

1. Clone the repository:

   git clone https://github.com/your-repo/zerovoice.git # Replace with actual repo URL if available
   cd zerovoice
   

2. Install dependencies:

   npm install
   

3. Start the development server:

   npm run dev
   

   This will typically start the application on http://localhost:5173 (or another available port).

4. Open in your browser: Navigate to the URL provided by Vite in your web browser.

5. Demo: You can try out a live demo here: https://biswatma.github.io/zerovoice/

Usage

• Once the application loads, ensure the "Loading..." message disappears and "Ready!" is displayed, indicating that all AI models have been loaded.
• Choose LLM Mode: Use the toggle switch to choose between:
  • Browser Mode (default): Uses the built-in SmolLM2-1.7B-Instruct model running in your browser
  • LM Studio Mode: Connects to a local LM Studio instance for more powerful models
• Click the "Start call" button to begin interaction. You may be prompted to grant microphone access.
• Speak naturally, and the AI assistant will respond in real-time.
• You can select different voices for the AI assistant using the dropdown menu.
• To end the conversation, click the "End call" button.

LM Studio Integration

To use LM Studio for more powerful language models:

1. Install LM Studio: Download and install LM Studio on your local machine.

2. Load a Model: In LM Studio, download and load any compatible language model (e.g., Llama, Mistral, etc.).

3. Start the Server: In LM Studio, go to the "Local Server" tab and start the server on the default port (1234).

4. Configure ZeroVoice:

   • Toggle the "LLM Mode" switch to "LM Studio" in the ZeroVoice interface
   • The application will automatically connect to LM Studio via a proxy (no additional configuration needed)

5. Start Conversation: Begin your conversation as usual. The application will now use your local LM Studio model for generating responses.

Note: LM Studio must be running and serving a model for the integration to work. If LM Studio is not available, the application will fall back to the browser-based model.

Linting

This project uses ESLint for code quality and consistency. You can run the linter using:

npm run lint