Text-To-Speech-VID

Automated pipeline for generating short-form video content (TikTok, YouTube Shorts, Instagram Reels) from JSON scripts. Combines AI-generated video clips, TTS narration, stock footage, and burned-in subtitles into ready-to-upload 9:16 vertical videos.

Pipeline Overview

content/*.json ──┬──> AI Video Clips (Wan 2.2 14B)  ──┐
                 │                                      ├──> Video Assembly ──> final_video.mp4
                 ├──> TTS Narration (Qwen3-TTS)  ──────┤
                 │                                      │
                 └──> Stock Footage (Pexels API)  ──────┘
                      (gap-filler only)

What it produces:

• 1080x1920 vertical video with AI-generated clips matched to narration
• Full TTS voiceover from a described voice (no reference audio needed)
• Word-by-word subtitles burned in, timed to speech
• Stock footage auto-fetched from Pexels to fill any gaps

Quick Start

1. Set up conda environments

The pipeline uses two separate environments due to incompatible dependencies:

# TTS environments
bash setup_envs.sh        # Linux/macOS
setup_envs.bat            # Windows

# AI video generation environment
setup_wan21.bat           # Windows (creates wan21 env with PyTorch + diffusers)

2. Install system dependencies

# ffmpeg (required for video assembly)
winget install ffmpeg      # Windows
brew install ffmpeg        # macOS
sudo apt install ffmpeg    # Ubuntu

# Base Python deps (in any env)
pip install requests python-dotenv tqdm

3. Set up Pexels API key (optional, for stock footage fallback)

# Create .env in project root
echo "PEXELS_API_KEY=your_key_here" > .env

Get a free key at pexels.com/api. Stock footage is only used when AI clips don't cover the full audio duration.

4. Generate everything

Two-step workflow:

# Step 1: Generate AI video clips (GPU-heavy, can run overnight)
conda activate wan21
python batch_ai_clips.py

# Step 2: Generate TTS + assemble videos + copy to output folder
conda activate qwen3-tts
python batch_videos.py

Or process a single video:

# AI clips for one content
conda activate wan21
python batch_ai_clips.py --only immortal_jellyfish

# TTS + assemble for one content
conda activate qwen3-tts
python batch_videos.py --only immortal_jellyfish

Finished videos are copied to D:\Shorts by default (configurable with --dest).

Project Structure

Text-To-Speech-VID/
├── content/                        # Content scripts (one per video)
│   ├── axolotl.json
│   ├── cuttlefish.json
│   ├── honey_badger.json
│   ├── immortal_jellyfish.json
│   ├── lyrebird.json
│   ├── mantis_shrimp.json
│   ├── octopus_intelligence.json
│   ├── pistol_shrimp.json
│   ├── platypus.json
│   └── tardigrade.json
├── voices/                         # Voice profile definitions
│   └── narrator_storyteller.json
├── tts_backends/                   # TTS model scripts (run in their conda envs)
│   ├── qwen_generate.py           #   Qwen3-TTS VoiceDesign backend
│   └── moss_generate.py           #   MOSS-TTS VoiceGenerator backend
├── video_gen/                      # AI video generation
│   ├── wan_generate.py             #   Wan 2.2 T2V-A14B GGUF clip generator
│   └── wan_test.py                 #   Single-clip test script
├── output/                         # All generated artifacts (git-ignored)
│   └── <content-id>/
│       ├── ai_clips/               #   AI-generated video clips
│       │   ├── ai_clip_00.mp4
│       │   ├── ai_clip_01.mp4
│       │   └── ...
│       ├── qwen3/                  #   TTS output
│       │   ├── full.wav            #   Complete narration audio
│       │   └── section_timings.json#   Per-section duration manifest
│       ├── footage/                #   Downloaded Pexels stock clips
│       └── final_video.mp4         #   Finished video ready to upload
│
├── batch_ai_clips.py              # Batch: AI clip generation (wan21 env)
├── batch_videos.py                # Batch: TTS + assembly + copy (qwen3-tts env)
├── batch_generate.py              # Legacy: all-in-one batch (single env)
├── generate.py                    # TTS orchestrator (dispatches to backends)
├── video_assemble.py              # Video assembly (clips + audio + subtitles)
├── audio_utils.py                 # WAV concatenation utility
├── requirements.txt               # Base deps: requests, python-dotenv, tqdm
├── setup_envs.bat                 # Windows env setup (qwen3-tts, moss-tts)
├── setup_envs.sh                  # Linux/macOS env setup
├── setup_wan21.bat                # Windows wan21 env setup
├── .env                           # Pexels API key (git-ignored)
└── .gitignore

Content Scripts

JSON files in content/ define each video:

{
  "id": "immortal_jellyfish",
  "title": "The Immortal Jellyfish",
  "voice": "voices/narrator_storyteller.json",
  "video_keywords": ["jellyfish", "jellyfish underwater", "jellyfish glowing"],
  "video_prompts": [
    "A tiny translucent Turritopsis dohrnii immortal jellyfish smaller than a fingernail glowing softly in dark deep ocean water, real nature documentary footage, macro lens, photorealistic, 4K",
    "..."
  ],
  "sections": {
    "hook": "Attention-grabbing opener...",
    "body": "Main fact with rich detail...",
    "closer": "Punchy ending line..."
  }
}

Voice Profiles

JSON files in voices/ describe the narrator voice style. Each TTS backend has its own config key:

{
  "name": "narrator_storyteller",
  "description": "A warm, confident male narrator with a deep, steady voice...",
  "qwen3": {
    "language": "English",
    "instruct": "A warm, confident male narrator..."
  },
  "moss": {
    "instruction": "A warm, confident male narrator..."
  }
}

AI Video Generation (Wan 2.2 14B GGUF)

Uses Wan 2.2 T2V-A14B with GGUF Q3_K_M quantization (~14.3 GB total for both experts). The model uses a Mixture of Experts architecture with separate high-noise and low-noise transformer experts.

Specs:

• Resolution: 480x832 (portrait 9:16), upscaled to 1080x1920 during assembly
• Frames: 81 (~3.4s at 24fps)
• Inference: 30 steps with dual guidance scales (4.0 / 3.0)
• Optimizations: FirstBlockCache (~2-3x speedup), CPU offload
• Time: ~15 min per clip on RTX 3090 (including warm-up)

Prompt tips for 14B model:

• Include specific animal descriptions (size, color, distinguishing features)
• Add "real BBC/National Geographic documentary footage, photorealistic"
• Use "macro lens" for small creatures
• 8 prompts per video works well (one clip per narration beat)

TTS Generation

Qwen3-TTS VoiceDesign (primary): Describe a voice in natural language — no reference audio needed. Generates the full script as one audio for voice consistency, then measures per-section timing ratios for subtitle alignment.

MOSS-TTS (secondary): Alternative model for A/B comparison.

The TTS orchestrator (generate.py) handles environment detection:

• If already in the target conda env, runs the backend directly
• Otherwise, finds conda and uses conda run -n <env>

Video Assembly

video_assemble.py combines everything into the final video:

1. AI clips are the primary video source (scene-matched to narration)
2. Stock footage from Pexels fills gaps if AI clips are shorter than audio
3. All clips are center-cropped and scaled to 1080x1920
4. Clips are trimmed to 2.5-4.5s random lengths for variety
5. Looped if needed to match audio duration
6. Subtitles are burned in as 3-word chunks timed to speech (uses section_timings.json from TTS if available)
7. Final encode: H.264 + AAC with faststart for streaming

Batch Scripts

batch_ai_clips.py (run in wan21 env)

Generates AI video clips for all content scripts. Skips content that already has all clips generated.

python batch_ai_clips.py              # all content
python batch_ai_clips.py --only axolotl  # single content

batch_videos.py (run in qwen3-tts env)

Generates TTS, assembles videos, and copies finished videos to the destination folder. Skips cached TTS and up-to-date videos.

python batch_videos.py                          # all content
python batch_videos.py --only honey_badger      # single content
python batch_videos.py --skip-tts               # only assemble + copy
python batch_videos.py --dest "D:\MyVideos"     # custom destination

Caching

Both batch scripts skip work that's already done:

• AI clips: Skipped if ai_clip_XX.mp4 exists for all prompts
• TTS: Skipped if full.wav exists and content hash matches (re-generates if script text changes)
• Video assembly: Skipped if final_video.mp4 is newer than both TTS audio and all AI clips

Individual Scripts

# Generate TTS only
python generate.py content/axolotl.json --models qwen3
python generate.py content/axolotl.json --models qwen3 --concat --video

# Concatenate section WAVs into full.wav
python audio_utils.py output/axolotl/qwen3

# Assemble video from existing clips + audio
python video_assemble.py content/axolotl.json --audio output/axolotl/qwen3/full.wav

# Generate AI clips for one content
python video_gen/wan_generate.py content/axolotl.json

# Test a single AI clip generation
python video_gen/wan_test.py

Requirements

• GPU: NVIDIA with CUDA support (tested on RTX 3090 24GB)
• Conda: Miniconda or Anaconda
• ffmpeg: Must be installed and on PATH
• Python: 3.11 (wan21), 3.12 (qwen3-tts, moss-tts)
• Pexels API key: Optional, only needed for stock footage fallback