AI Voice Clone Studio

Self-hosted zero-shot voice cloning and text-to-speech for Urdu (Perso-Arabic + Roman) and English. Web app — FastAPI backend, React frontend, open models running on your own GPU. No API keys, no external services, no per-character billing.

Status: validated end-to-end on GPU with real cloned audio. VoxCPM 2 (English + Roman Urdu) and OmniVoice (native Urdu script) are the deployed models; an async job queue, Speech Direction, a pronunciation dictionary, a Convert tab (paste a script, convert its writing system), and a Roman/Devanagari→Urdu transliterator (Gemma-4-31B) have all landed. Hindi was fully removed as a target language — it survives only as a source format: paste a Devanagari script into the Convert tab and it transliterates to Urdu, never spoken as Hindi. Design and rationale: docs/REWRITE_PLAN.md. Architecture: docs/ARCHITECTURE.md. Working agreements for contributors and agents: CLAUDE.md. What's done / in progress / planned next: docs/ROADMAP.md.

Base branch is main, which the rewrite was merged into on 2026-08-06.

Languages

You declare the language; the app detects the script. It cannot guess Roman Urdu from English — "Aap kaise hain" and "How are you" are both Latin — so it does not try. Every result carries a visible chip naming the model and any transformation applied, including whether it was lossy.

Two deployed models cover it. VoxCPM 2 is tokenizer-free and renders romanized text directly, so Roman Urdu needs no transliteration step to be routable. OmniVoice serves native Urdu script (اردو), verified 2026-08-15:

Cloning is cross-lingual: the reference speaker need not have spoken the target language.

Roman Urdu is routable but sounds accented. VoxCPM 2 renders "aap kaise hain" as valid audio, but a native listener hears an English accent (measured finding A0). So the app offers a Gemma-4-31B transliterator that converts Roman Urdu → Urdu script, which OmniVoice reads properly — and it's never silent: it shows you the converted text to check first, because the model's failure mode is a real Urdu word that means something else. In the Composer this happens on Generate (one tap to confirm); in the Convert tab you paste a script and convert it per-part.

You declare the language; the app detects the script — it never guesses. If no model can render what you asked for, you get a 422 listing what would work, never substituted audio. Every result carries a visible chip naming the model and any transformation applied.

Models

Permissive by default; CC-BY-NC allowed for the owner's own use, badged "Non-commercial". No paid tiers, ever. Each model's license is checked against its HF card separately from the repo's code license.

The transliterator and analyzer are not voice runtimes and are deliberately unreachable from routing — they transform text, which a human reviews before it's ever spoken. XTTS v2 (CPML) and Fish Speech (research license) stay banned as non-commercial; Higgs Audio v3 (research-only) is excluded even more strictly than CC-BY-NC.

Model capability claims are not taken from README files. Each (model × language × script) pair must pass a measured gate — CER < 25 %, speaker similarity > 0.70, faster than realtime, and a human listen — or it is removed from the catalog rather than advertised.

Requirements

• NVIDIA GPU, 12 GB+ VRAM for VoxCPM 2 alone (developed on a 24 GB RTX A5000). CPU inference is not supported. The optional Gemma-4-31B transliterator adds ~19 GB (4-bit) — it's resident and idle-killed, and takes the whole GPU slot while converting rather than co-residing with the voice models, so it fits a 24 GB card but leaves little slack. Skip it and everything except Roman→Urdu-script conversion still works.
• Python 3.12 and uv
• Node 20+
• ffmpeg on PATH

Quick start

The design puts the GPU model in a separate OS process with its own interpreter, so setup has two environments: the API venv (no torch, by design) and one runtime venv per model. Getting audio out requires both — starting only the API gives you a server whose every /generate fails with "no interpreter for runtime voxcpm".

git clone https://github.com/MunawarAliAraiz/AI-Voice-Clone.git
cd AI-Voice-Clone

1. API environment — deliberately contains no torch:

cd backend && uv sync

2. VoxCPM 2 runtime environment — a second venv that does have torch. On a fresh GPU box, scripts/pod-bootstrap.sh does all of this for you (see Cloud / RunPod below). By hand:

uv venv backend/.venv-voxcpm --python 3.12
uv pip install --python backend/.venv-voxcpm voxcpm
# Pin torch to the CUDA build your driver supports. A plain `pip install voxcpm`
# pulls a cu130 wheel that silently reports cuda.is_available()==False on older
# drivers, then runs on CPU and times out. Check your driver's max CUDA with
# `nvidia-smi`; cu128 fits driver 525+ / CUDA 12.8:
uv pip install --python backend/.venv-voxcpm \
  torch==2.8.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128
# verify — this MUST print True:
backend/.venv-voxcpm/bin/python -c "import torch; print(torch.cuda.is_available())"

3. Start the API, pointed at that runtime:

cd backend
export HF_HOME=/path/to/persistent/hf-cache          # or several GB re-download every restart
export VCS_VOXCPM_PYTHON="$PWD/.venv-voxcpm/bin/python"
export VCS_WORKER_CWD="$PWD"
uv run uvicorn app.main:app --host 0.0.0.0 --port 8000

The first /generate of a process spawns the worker and downloads/loads the weights (~7 GB, once); after that it is roughly real-time. Run one uvicorn worker — N workers means N schedulers, each believing it owns all your VRAM.

4. Frontend:

cd frontend && npm install && npm run dev      # http://localhost:1420

The dev server proxies /api to http://localhost:8000 by default. To point it at a backend elsewhere — e.g. a GPU pod reached over an SSH tunnel — set VITE_PROXY_TARGET: VITE_PROXY_TARGET=http://127.0.0.1:8010 npm run dev. This stays same-origin (the browser only talks to the vite server), so CORS never applies. There is no in-app setting for the backend address — see Deploying for why.

No-GPU smoke test

To exercise the full stack on a machine with no GPU, enable the gated silence runtime — it returns silent audio with a loud X-Fake-Audio: true marker, never a fallback masquerading as real output:

cd backend && VCS_ALLOW_FAKE_RUNTIME=1 uv run uvicorn app.main:app --port 8000

Cloud / RunPod

scripts/pod-bootstrap.sh rebuilds a fresh GPU pod from zero — clones the repo, redirects caches off the ephemeral overlay, and builds the API venv plus every runtime venv (VoxCPM 2, OmniVoice, Chatterbox, the Qwen analyzer, and the Gemma transliterator) with the cu128 torch pin and pinned weights. It also writes /workspace/ctl.sh, a one-command backend lifecycle (ctl.sh up|start|restart|stop|status). The repo is public for read, so no token is needed. One command, from your laptop:

ssh -p PORT -i ~/.ssh/id_ed25519 root@HOST "bash -s" < scripts/pod-bootstrap.sh

-i ~/.ssh/id_ed25519 points at your private key explicitly — use it if you're not relying on an ssh-agent to offer the key automatically. Piping the script in over "bash -s" < scripts/... isn't backgrounded, so every line the script prints (uv sync, the torch install, the weight download, the test run) streams to your terminal live as it happens — there's nothing extra to do to see the logs. On Windows, use C:\Windows\System32\OpenSSH\ssh.exe if your key is passphrase-protected and loaded in the Windows ssh-agent; Git Bash's ssh can't see that agent.

Add START=1 and it also starts the backend and ngrok and polls until /api/health answers, so the one command is the whole deployment. It ends by printing your VCS_API_KEY in full and a status block showing backend / ngrok / public-URL health.

FRONTEND_URL and NGROK_DOMAIN are saved to /workspace on first use and reused afterwards, so a restart on the same volume only needs START=1 NGROK_AUTHTOKEN=…. To reach the API from your laptop without a public URL, forward the port over SSH instead: ssh -N -L 8000:127.0.0.1:8000 -p PORT root@HOST.

Moving to a brand-new pod

A pod restart keeps /workspace; a genuinely new volume does not. Pass your existing secrets so the new pod adopts them and no browser has to be re-paired:

ssh root@NEW_HOST -p NEW_PORT "START=1 NGROK_AUTHTOKEN=<token> \
  NGROK_DOMAIN=<your-name>.ngrok-free.dev FRONTEND_URL=https://<your-app>.workers.dev \
  VCS_API_KEY=<existing> VCS_MEDIA_TOKEN_SECRET=<existing> bash -s" < scripts/pod-bootstrap.sh

Both secrets or neither — the script refuses a half pair. Supplying only the API key would silently regenerate the media secret and 403 every audio URL already handed out. Omit both and it generates a new pair and tells you to re-paste the key.

Deploying (frontend on Cloudflare, backend on a GPU pod)

The frontend deploys to Cloudflare Workers, and frontend/worker.js serves the static assets and proxies /api/* to the pod's ngrok tunnel. The proxy is not incidental: an <audio> element cannot send a custom header, so it could not carry the ngrok-skip-browser-warning that fetch uses to get past ngrok's free-tier interstitial, and every clip came back as an HTML warning page that the browser failed to decode. Proxying makes the API same-origin, so the header is injected server-side — and CORS drops out of the picture entirely.

Consequences worth knowing:

• Do not set VITE_API_BASE in the Cloudflare build. Same-origin is already the production default; setting it sends requests cross-origin again and reinstates the bug.
• There is no backend-address setting in the app. Only an API key field. A saved override outlives a deploy and breaks exactly one device while every other one works.
• BACKEND_ORIGIN in frontend/wrangler.toml is the one place the tunnel URL lives. Update it there if the ngrok domain changes.

docs/POD_SETUP.md has the rest: resuming after a pod restart, connecting a local frontend, the manual steps for when something needs debugging, and how to diagnose audio that plays on a laptop but not a phone.

Configuration

Environment variables, prefix VCS_, from backend/.env (host/port are uvicorn CLI flags, not settings):

There is no default_engine setting. Routing decides per request from the declared language and the detected script.

Tests

cd backend && uv run pytest              # CPU-only, no torch, ~30s
cd backend && uv run pytest -m gpu       # real weights, real GPU
cd frontend && npm run build      # tsc -b + vite build (no test script yet)

The scheduler tests — including the ones covering concurrent load and eviction — need no GPU.

Licensing and consent

The code is one thing; the weights are another. Every model here is permissively licensed, but CC-BY-SA-4.0 (OpenBible Urdu) requires attribution — see NOTICE.

Only clone voices that are your own or that you have the speaker's explicit consent to use. Voice recordings are biometric data in a number of jurisdictions.