VoiceMatch Labs

AI-powered English pronunciation coaching — analyze speech at the phoneme and pitch level, then visualize it against a native-speaker reference.

VoiceMatch is a full-stack, cloud-native application that helps learners improve their English pronunciation through fine-grained speech analysis and interactive visual feedback. Users record themselves (or pull a reference clip from YouTube), and the system breaks the audio down into words, phonemes, and a pitch/intonation contour, then renders the result on an interactive chart so they can compare their pronunciation against a native speaker and practice deliberately.

This is a monorepo containing the web frontend, a serverless backend, and a self-contained ML model-serving toolkit (voicematch-models).

---

Table of Contents

1. Overview
2. Architecture
3. Tech Stack
4. Features
5. Repository Layout
6. Getting Started
7. Links

---

Overview

Pronunciation feedback tools usually stop at "right / wrong." VoiceMatch goes deeper: it aligns recognized words and phonemes to their timestamps, estimates the speaker's pitch contour, and overlays everything so a learner can see exactly which sounds and which intonation patterns drift from a native reference.

The platform is split into three cooperating components:

• VoiceMatch Web — a Vue 3 single-page app for recording, importing reference clips, and visualizing phoneme- and pitch-level feedback.
• VoiceMatch Backend — a serverless (AWS Lambda) API that ingests/ trims audio, fans requests out to the ML services, and fuses the results (word/phoneme alignment + interpolated pitch contour) into a single response for the frontend.
• VoiceMatch Models — a model-serving toolkit packaging three inference services (word recognition, phoneme recognition, pitch evaluation) as containers for TorchServe / AWS SageMaker.

Phoneme-level breakdown and pitch contour for a spoken English phrase.

Architecture

A recording (or a YouTube reference clip) is uploaded to S3, trimmed and transcoded server-side, then fanned out in parallel to three ML inference services. The backend fuses word/phoneme timing with an interpolated pitch contour and returns it to the frontend for visualization. The model services run either on AWS SageMaker (cloud) or locally as Docker containers (the same images, swapped by endpoint).

flowchart TD
    user([User / Browser + Mic])
    web["VoiceMatch Web<br/>(Vue 3 SPA)"]
    yt[(YouTube reference clip)]
    s3[(AWS S3<br/>recordings & downloads)]
    backend["VoiceMatch Backend<br/>(AWS Lambda / Serverless)"]

    subgraph models["VoiceMatch Models (SageMaker / Docker)"]
        word["Word recognition<br/>Wav2Vec2 · PyTorch"]
        phon["Phoneme recognition<br/>Wav2Vec2 · PyTorch"]
        pitch["Pitch evaluation<br/>SPICE · TensorFlow"]
    end

    user -->|record / import| web
    web -->|upload audio| s3
    yt -->|/youtube/download| backend
    web -->|/audio/process| backend
    backend <-->|fetch / trim| s3
    backend -->|audio stream| word
    backend -->|audio stream| phon
    backend -->|audio stream| pitch
    word --> backend
    phon --> backend
    pitch --> backend
    backend -->|words + phonemes + pitch contour| web
    web -->|interactive chart| user

Request flow

1. The user records audio in the browser (or imports a native reference clip from YouTube).
2. Audio is stored in S3; the backend trims/transcodes the requested segment (FFmpeg) on demand.
3. The backend streams the audio to three inference services in parallel.
4. Word and phoneme models (Wav2Vec2) return timestamped tokens; the pitch model (SPICE) returns a confidence-weighted pitch/semitone contour.
5. The backend interpolates pitch onto phoneme boundaries and returns a unified payload.
6. The frontend renders an interactive chart for the learner to analyze and practice against.

Tech Stack

Features

• End-to-end speech analysis — word recognition, phoneme recognition, and pitch estimation from a single recording.
• Native-reference comparison — import reference clips straight from YouTube to practice against.
• Phoneme/pitch fusion — pitch contour interpolated onto phoneme boundaries for per-sound intonation feedback.
• Interactive visualization — Plotly-based charts with playback controls (Tone.js / Web Audio).
• In-browser recording — extendable-media-recorder + AudioWorklets, with client-side WAV/Opus encoding.
• Parallel model serving — three independent inference services callable locally (Docker) or on SageMaker.
• Serverless backend — Lambda functions for YouTube info/download and audio processing, with on-demand S3 caching and FFmpeg trimming.

Repository Layout

voicematch-labs/
├── voicematch-web/        # Vue 3 SPA — recording, import, and visualization (primary frontend)
├── voicematch-backend/    # Serverless (AWS Lambda) API: audio ingest, trimming, model orchestration
│   └── src/
│       ├── handler/       # Lambda handlers: youtubeInfo, youtubeDownload, audioProcess
│       ├── services/      # predictions.ts — calls to the three ML inference services
│       └── utils/         # s3, ffmpeg, youtube-dl helpers
├── voicematch-models/     # Model-serving toolkit (Docker / TorchServe / SageMaker)
│   └── models/
│       ├── wordrecog/     # Wav2Vec2 word recognition
│       ├── phonerecog/    # Wav2Vec2 phoneme recognition
│       └── pitcheval/     # SPICE pitch evaluation
├── notebooks/             # Research notebooks (HF transformers, Wav2Vec2, pitch estimation)
├── voicematch-research/   # Exploratory model/training scripts
├── langchain-research/    # Unrelated LangChain experiments
└── web/                   # Frontend scaffolds and Web Audio / worklet experiments

Note: voicematch-models is documented separately and linked from the portfolio. See its README for model details and serving instructions.

Getting Started

Each component is self-contained. See the per-component READMEs for full instructions: voicematch-web, voicematch-backend, voicematch-models.

Prerequisites

• Node.js ≥ 18 and npm
• Python 3.8+ and Docker (for the ML services)
• FFmpeg on PATH
• AWS credentials (for S3 / SageMaker; optional for fully local runs)

1. Run the ML inference services

# Build an inference image (example: word recognition)
cd voicematch-models/models/wordrecog
docker build -t voicematch-wordrecog:latest .

# Serve it (repeat for phonerecog and pitcheval on their own ports)
docker run --rm -d --name voicematch-wordrecog -p 7081:8080 voicematch-wordrecog:latest serve

The backend expects the three services on localhost:7081 (words), localhost:7082 (phonemes), and localhost:7083 (pitch). See voicematch-backend/src/services/predictions.ts.

2. Run the backend (Serverless Offline)

cd voicematch-backend
npm install
cp .env.example .env   # fill in AWS region / bucket values
npm run debug          # serverless offline on http://localhost:3000

Endpoints: GET /youtube/info, GET /youtube/download, GET /audio/process.

3. Run the frontend

cd voicematch-web
npm install
npm run dev            # Vite dev server

Links

• Portfolio: iliazlobin.com/portfolio
• Author: Ilia Zlobin — Principal Software Engineer

License

Released under the MIT License.