🌸 Sakura Subtitle Generator

A powerful, well-architected application for generating Japanese subtitles with multi-language translation support. Built with modern AI models and a user-friendly Tkinter interface. Get the highest quality subtitles locally.

✨ Features

• 🎙️ Japanese ASR: High-quality Japanese speech recognition using OpenAI Whisper
• 🌐 Multi-language Translation: Translate to English and Traditional Chinese using Hugging Face models
• 🎯 Dual Language Subtitle Output: Generate both original Japanese and translated subtitle files with customizable file name suffixes
• 🖥️ User-Friendly GUI: Clean Tkinter interface for easy video processing
• ⚙️ Configurable: Comprehensive settings for models, devices, and output preferences
• 🏗️ Well-Architected: Modular design with clean separation of concerns
• 📊 Progress Tracking: Real-time progress updates during processing
• 💾 Multiple Export Formats: SRT subtitle export with more formats planned
• ⚡ GPU-First Design: Optimized for NVIDIA CUDA and Apple Silicon MPS acceleration

🆕 Recent New Features (Dec 5, 2025) - ASR Pipeline Overhaul & Translation Upgrade

🎙️ ASR Pipeline Overhaul: "Strict Timestamp" Architecture

We have completely re-architected the ASR pipeline to prioritize timestamp accuracy and hallucination prevention, addressing the "2-minute segment" and "silence hallucination" issues.

Key Changes:

1. Strict Timestamp Adherence:
   • Removed: The JapaneseLinguisticProcessor merging logic, which previously ignored timestamps to find "sentence boundaries" (causing 2-minute segments).
   • Result: Subtitles now strictly follow Whisper's granular timestamps (e.g., 1.5s, 3.2s), ensuring perfect sync with audio.
2. Silence Hallucination Prevention:
   • New Logic: Implemented max_merge_gap (2.0s).
   • Mechanism: If two speech segments are separated by >2s of silence, they are never merged. This prevents sending long silence blocks to Whisper, eliminating gibberish hallucinations (e.g., "法斯米奇鸡汁").
3. VAD Merge Fixes:
   • Correction: Fixed logic to correctly account for silence gaps when calculating chunk duration, ensuring chunks never exceed the 30s target.
   • Timestamp Fix: Fixed a critical bug in split_audio_chunks that was resetting timestamps to 0, ensuring absolute timing is preserved.

🌸 Translation & Output Upgrades

1. New Flagship Model:
   • Model: Sakura-14B-Qwen2.5-v1.0-GGUF (Standard 4-bit)
   • Improvement: Significantly smarter and more coherent than previous quantized versions.
2. Performance Optimization:
   • Batched Retries: Failed translations are now grouped into a single batch retry instead of sequential retries, speeding up recovery.
3. Readability:
   • Text Wrapping: Added automatic text wrapping (40 characters) for SRT export, ensuring long sentences are readable on screen.

🆕 Recent New Features (Dec 2, 2025) - Hallucination Prevention & Quality Improvements

🛡️ Comprehensive Hallucination Prevention Pipeline

A sophisticated multi-stage pipeline has been implemented to eliminate LLM hallucinations and drastically improve translation quality:

VAD Merging (1888→230 segments)
    ↓
SakuraLLM Batch Translation
    ↓
_detect_hallucinations() Detection (repetition > 30%)
    ↓
confidence=0.0 Marking
    ↓
HallucinationFilter Filtering
    ↓
✨ Clean Output

Key Improvements:

📊 Quality Metrics (Validation Results)

Processing 61-minute video with 195 subtitle segments:

Hallucination Reduction:

• 「好吃」(repeated "delicious"): 100% eliminated ✅
• 「嗯」(filler sound) reduction: 8-10% decrease
• 「請請」(repeated "please"): 100% eliminated ✅
• Overall hallucination filter effectiveness: 7 entries removed → 188 clean translations

Structural Improvements:

• Chinese subtitle lines: 759 → 751 lines (8-line reduction from filtering)
• Chinese subtitle entries: 379 → 375 entries (4-entry reduction from hallucination removal)
• Time span improvement: 16-17s fragments → 24-42s merged segments (contextually richer)

Content Quality:

🔧 Configuration & UI Improvements

Exclusive Chunk Control:

• chunk_length (direct split): Used only when VAD disabled (5-60s, default 9s)
• chunk_length_s (VAD merge target): Used only when VAD enabled (5-30s, default 14s)
• Settings UI now mutually exclusive - enabling one automatically disables the other

Enhanced Settings Dialog:

• Grid layout fixes (removed float row indices causing "bad row value" errors)
• Separate controls for non-VAD vs VAD chunking strategies
• Real-time enable/disable based on VAD toggle

✨ Technical Implementation

New Methods Added:

• _detect_hallucinations(): Detects repetitive character patterns and marks as low-confidence
• Enhanced _on_vad_toggled(): Manages mutually exclusive chunk controls
• Improved _apply_hallucination_filter(): Logs detailed filtering operations

Configuration Parameters:

{
  "asr": {
    "chunk_length": 9,        // Direct split threshold (non-VAD)
    "vad": { "enabled": true }
  },
  "chunk_length_s": 14        // VAD merge target (soft limit)
}

🆕 Recent New Features (Dec 1, 2025)

• SakuraLLM Official Prompt Format: Implemented SakuraLLM's official prompt format with system/user message structure and glossary support. Custom terminology dictionaries can be placed in prompts/ directory for specialized translation needs.
• LLM Hallucination Prevention: Added line number range filtering to prevent LLM from generating extra content beyond the expected batch range, improving translation alignment accuracy.
• Enhanced ASR Sensitivity Settings: New configurable parameters for better speech recognition of quiet audio:
  • no_speech_threshold: Lowered to 0.3 (default 0.6) to capture more low-volume speech
  • preamp: Increased to 6.0 for audio amplification
  • condition_on_previous_text: Disabled to reduce repetition
  • compression_ratio_threshold: Set to 3.0 for more flexible content acceptance
• Expanded ASR Model Selection: Added japanese-asr/distil-whisper-large-v3-ja-reazonspeech-all to UI model selector, providing a fast and memory-efficient Japanese ASR alternative (~1.5GB).
• Full GPU Offload for 14B Models: With optimized small prompts, SakuraLLM 14B now runs with n_gpu_layers=-1 (full GPU offload) on Apple Silicon, eliminating previous OOM issues.

🆕 Recent New Features (Nov 30, 2025)

• SakuraLLM 14B Apple Silicon Optimization: Automatic detection of 14B models on Metal with partial GPU offload (20 layers) and optimized context window (6144 tokens) to prevent OOM errors while maximizing performance.
• Dynamic Batch Size Calculation: Batch size now calculated based on actual prompt size and available context, increasing from 4-6 to 15 segments per batch with optimized prompts.
• Retry Mechanism for Untranslated Segments: Post-processing detects segments still containing Japanese (hiragana/katakana) and retries them individually for better translation coverage.
• Enhanced Batch Parser: Improved handling of LLM output formats including Original> with Chinese content, Translation>/翻譯> prefixes, and alignment preservation for failed translations.
• VAD Fallback UI Improvements: Fixed VAD queue display (now shows cumulative count instead of always-zero queue size) and reset behavior between jobs.

🆕 Recent New Features (Nov 2025)

• VAD Improvements & UI Controls: WebRTC VAD tuning updated (default frame_duration_ms=30) and VAD parameters (enable, mode, frame duration, padding, min segment duration) exposed in the Settings dialog for easy runtime tuning.
• Whisper ASR stability: Suppressed a deprecation warning related to return_token_timestamps while keeping native Whisper generate() usage for reliable timestamps and no token-limit behavior.
• Modular Prompt Templates: Prompt templates are now stored under prompts/; the SakuraLLM translator loads the selected template from config. The Settings dialog dynamically lists templates available in prompts/.
• SakuraLLM Prompt Flow Fixed: SakuraTranslator.create_from_config() now passes the configured prompt template style to the translator so the chosen template is actually used during translation.
• UI & Config Fixes: Settings dialog layout fixes, persistent config saving/loading for new options, and safer default fallbacks.
• Batch Translation Optimization: Enhanced SakuraLLM batch translation parser to handle multiple prompt formats (structured with Translation> prefixes and simple line-by-line), enabling ~20x faster translation while preserving timestamps and supporting all existing prompt templates.

🐞 Bug Fixes (Nov 2025)

• VAD zero-interval issue: Resolved the case where VAD reported 0 speech intervals by enforcing supported WebRTC frame durations (10/20/30 ms) and defaulting to 30 ms for long-form content.
• Prompt template selection bug: Fixed translator initialization so non-standard templates selected in the UI are applied correctly.
• Syntax & compile fixes: Cleaned stray syntax errors and verified modules compile with python -m py_compile.
• Batch Translation Parsing: Fixed critical SakuraLLM batch response parser to intelligently detect output format (structured vs line-by-line) based on Translation> prefix ratio, preventing ~90% translation loss where translations were replaced with Japanese text.
• Batch Translation Parsing: Fixed SakuraLLM batch response parser to handle both structured format (with Translation> prefixes) and simple line-by-line format, resolving Chinese translation filtering issues where 541 translations were incorrectly rejected.

🆕 Recent Features (Nov 28, 2025)

• VAD resilience (hybrid fallbacks): Per-scene quick VAD sweeps are now performed when default VAD returns no intervals. The pipeline falls back to an offline exhaustive VAD report when available and, as a last resort, uses a full-scene fallback so scenes are never skipped.
• Preserve absolute timestamps for ASR: ASR now uses absolute AudioData.start_time across scene-based processing so subtitle timestamps remain correct after chunking.
• Audio pre-amplification (auto + manual): Added configurable preamp with an automatic heuristic (asr.auto_preamp_enabled, asr.target_rms, asr.preamp_max) and manual override (asr.preamp) exposed in the Settings dialog to improve VAD and ASR quality on low-RMS recordings.
• Background VAD fallback recorder: Per-scene fallback decisions are persisted asynchronously to outputs/vad_fallbacks.jsonl via a background writer to avoid blocking IO during processing.
• UI visibility for VAD fallbacks: New VAD fallbacks dialog and a small VAD queue indicator were added to the main window so you can inspect fallback records and recorder status during runs.
• Japanese ASR postprocessing: Whisper ASR integrates Japanese-specific linguistic postprocessing (ending clusters and fillers) to improve segment merging and subtitle readability.
• Translation lazy-loading & default change: Translators (SakuraLLM and multi-stage pipeline) are now lazily loaded only when requested by target languages. The default target languages were changed to ["zh"] (Chinese) and English translation is opt-in in the UI.

🎯 GPU Acceleration Focus

This project is GPU-first and optimized for hardware acceleration:

✅ Supported GPU Acceleration:

• 🚀 NVIDIA CUDA: Full GPU acceleration for RTX/GTX cards
• 🍎 Apple Silicon MPS: Metal Performance Shaders for M1/M2/M3/M4 chips
• ⚡ Automatic Detection: Intelligent GPU selection (CUDA > MPS > CPU)

⚠️ CPU-Only Users:

If you only have CPU and need CPU-optimized inference, this project focuses on GPU acceleration. For CPU-only setups, consider:

• Alternative: Implement ctransformers-based classes for CPU optimization
• Current Focus: This project prioritizes GPU performance (MPS/CUDA)
• Performance: GPU provides 3-5x faster inference than CPU-only solutions
• Development: CPU-specific optimizations (ctransformers, GGUF, etc.) are not the main focus

Recommended Hardware:

• Apple Silicon Macs: M1/M2/M3/M4 with 16GB+ RAM
• NVIDIA GPUs: RTX/GTX cards with 8GB+ VRAM
• Minimum: 16GB system RAM for model loading

🚀 Cache System & VAD Optimization (Dec 2, 2025)

💾 Intelligent LRU Cache System

Processing large videos repeatedly no longer re-extracts audio or re-detects scenes:

Cache Features:

• Checksum-Based Keys: Files identified by MD5 hash of content (not path), enabling seamless file portability
• Dual Caching: Both audio WAV files AND scene detection metadata cached separately
• LRU Eviction: Automatic cleanup when cache exceeds 10 items (balances storage vs. hit rate)
• Smart Invalidation: Cache automatically invalidates if video file is modified

Cache Structure:

.cache/
├── audio/              # Extracted WAV files
│   └── video_6d661ee2a9a1.wav
├── scenes/             # Scene detection JSON
│   └── video_6d661ee2a9a1_scenes.json
└── cache_metadata.json # LRU tracking + checksums

Performance Impact (3667s video test):

🎯 Optimized VAD Parameters (LOCKED)

Final configuration after extensive testing and mathematical validation:

{
  "vad": {
    "mode": 3,                          // Most sensitive detection
    "padding_ms": 300,                  // 0.3s boundary protection
    "min_segment_duration": 0.65,       // Filters ≤0.05s noise
    "frame_duration_ms": 30              // Standard 30ms frames
  }
}

Mathematical Validation:

Boundary protection: 2 × 300ms = 600ms
Min duration threshold: 650ms

Interpretation:
✅ Segments ≥ 0.65s preserved (speech)
✅ Segments < 0.05s filtered (noise)
✅ No clipping due to: 600ms < 650ms

Quality Impact:

• Reduced repetition: "シャキン" repeats from 5× → 1-2×
• Improved coherence: Complete sentences instead of fragments
• Unchanged ASR accuracy: Maintained 70-80% baseline
• Example outputs:
  • ✅ "ごめんなさい" (complete apology)
  • ✅ "めっちゃかわいい" (maintained excitement)
  • ✅ "ちょっと待ってください" (complete request)

Known Limitation:

• 30s fixed intervals persist in some Whisper chunks (internal Whisper tokenization, external to our control)
• Documented as acceptable trade-off vs. quality impact

📊 Combined System Performance

Full pipeline on Apple Silicon M4:

🔧 Implementation Details

New File: src/utils/cache_manager.py

• Checksum generation: hashlib.md5(file_content).hexdigest()[:12]
• Cache eviction: Tracks last_accessed timestamps
• Atomic operations: Safe concurrent access via file locking

Integration Points:

• src/subtitle/subtitle_generator.py: Cache audio/scenes extraction
• src/utils/audio_processor.py: Load cached WAV files
• config.json: VAD parameters locked for reproducibility
• .gitignore: Cache directory excluded from version control

Bug Fixes (This Session):

• ✅ SceneSegment import: video_data → scene_detector
• ✅ Parameter naming: confidence → score (3 occurrences)
• ✅ File paths: video_file.path → video_file.file_path (3 occurrences)
• ✅ VAD logger syntax error fixed
• ✅ Empty scenes path fallback implemented

---

🚀 Hardware Acceleration Performance

Cross-Platform Hardware Acceleration with significant performance improvements:

⚡ Audio Extraction Performance:

🎯 ASR Processing Performance:

Hardware Acceleration Features:

• 🏁 Cross-Platform: VideoToolbox (Apple), CUDA (NVIDIA), Software fallback
• 🔥 Automatic Detection: Intelligent hardware capability detection
• ⚡ Audio Extraction: Up to 18.8x realtime performance on Apple Silicon
• 💚 Universal Compatibility: 100% fallback support for all hardware
• 🧠 Memory Efficient: Optimized resource management and cleanup

🚀 Quick Start

Hardware Requirements

🎯 Recommended (GPU Acceleration):

• Apple Silicon: M2/M3/M4 with 16GB+ unified memory
• NVIDIA GPU: RTX 3070/4070+ with 8GB+ VRAM
• System RAM: 16GB+ for model loading
• Storage: 10GB+ free space (3.6GB models + cache)
• Expected Performance: 8-12x realtime processing

💰 Minimum (Budget Setup):

• Apple Silicon: M1 with 8GB unified memory
• NVIDIA GPU: GTX 1660 with 6GB VRAM
• Alternative: Use whisper-medium (1.5GB vs 3GB)
• Expected Performance: 4-6x realtime processing

⚠️ CPU-Only (Not Optimized):

• Performance: 4-12x slower than GPU (1-2x realtime)
• Memory: 32GB+ RAM recommended for all models
• Models: Consider whisper-small for better CPU performance
• Note: Implement ctransformers for CPU optimization (outside project scope)

📝 See Default Models & Requirements section for detailed specifications

Software Prerequisites

• Python 3.8 or higher
• FFmpeg (for audio extraction)
• GPU drivers (CUDA 11.8+ for NVIDIA, latest macOS for Apple Silicon)

Installation

Method 1: Pure uv (Recommended - No Environment Pollution)

git clone <repository-url>
cd py-sub-generator-sakura

# No installation needed - runs in isolated environment!
uv run python main.py

• 🔒 Zero environment pollution - Completely isolated
• ⚡ Fastest startup - Auto-manages dependencies
• 🛡️ Safe - No system Python modification
• 🧹 Clean - No installation artifacts

Method 2: Universal Setup

git clone <repository-url>
cd py-sub-generator-sakura

uv run python setup.py  # Use uv run to avoid polluting system Python!
# OR (if you must):
python setup.py  # ⚠️ May install to system Python if uv not available

• ✅ Works on all platforms (Windows, Linux, macOS)
• ✅ Auto-detects uv for faster installation
• ✅ Checks system requirements (Python, FFmpeg)
• ⚠️ Warning: May pollute system Python if uv unavailable

Method 3: Apple Silicon Optimized (M1/M2/M3 Macs)

git clone <repository-url>
cd py-sub-generator-sakura

python setup_apple_silicon.py      # Maximum performance setup

• 🍎 Apple Silicon specific optimizations
• ⚡ 3-5x faster processing with MPS acceleration
• 🔧 Automatic FFmpeg installation via Homebrew
• 💾 20-30% less memory usage

Method 4: Manual Installation (Advanced)

git clone <repository-url>
cd py-sub-generator-sakura

# Install with uv (automatically handles dependencies)
uv pip install -e .                    # Basic installation
uv pip install -e ".[gpu]"            # With GPU support  
uv pip install -e ".[apple-silicon]"  # Apple Silicon optimized
uv pip install -e ".[dev]"            # Development dependencies

# Or install development dependencies separately
uv pip install -e . --group dev

Method 5: Traditional pip Installation (⚠️ Not Recommended)

git clone <repository-url>
cd py-sub-generator-sakura

pip install -e .                      # Basic installation
pip install -e ".[gpu]"              # With GPU support
pip install -e ".[apple-silicon]"    # Apple Silicon optimized

Usage

GUI Mode (Recommended)

# With uv (fastest, automatic dependency management):
uv run python main.py

# After installation with setup.py or setup_apple_silicon.py:
python main.py

CLI Mode (Future)

# With uv (no installation needed)
uv run python main.py --no-gui video.mp4

# After installation
python main.py --no-gui video.mp4

🎯 Dual Language Subtitle Feature

🌸 Generate Both Japanese and Translated Subtitles

The application supports dual language subtitle output, perfect for language learning and multilingual content creation:

✨ Feature Highlights:

• 🎌 Japanese Original Subtitles: Preserves original ASR transcription results
• 🌐 Translated Subtitles: Simultaneously generates target language (e.g., Chinese, English) subtitles
• ⚙️ UI Controls: Convenient toggle option in settings dialog
• 📝 Custom File Naming: Configurable file suffixes (e.g., _ja.srt, _zh.srt)
• 🎯 Flexible Output: Choose to generate single or dual language subtitles as needed

🛠️ How to Use:

Via GUI Settings:

1. Click the ⚙️ Settings button
2. Find the "Dual Language Output" option in the settings dialog
3. Check "Generate both language subtitles"
4. Optional: Click "Advanced Options" to customize file suffixes
5. Settings are automatically saved

File Output Example:

video.mp4 → 
├── video_ja.srt    (Japanese original subtitles)
└── video_zh.srt    (Chinese translated subtitles)

Advanced Customization:

• Japanese Suffix: Default _ja (can change to _japanese, _orig, etc.)
• Translated Suffix: Default _zh (can change to _chinese, _trans, etc.)
• File Naming: Fully customizable for workflow integration

🎬 Perfect Use Cases:

• 🎓 Language Learning: Compare original and translated text for learning
• 📺 Multilingual Content: Provide options for different audiences
• 🔄 Translation Comparison: Check translation quality and accuracy
• 📚 Subtitle Production: Professional subtitle creation workflows

Development with uv

# Install dev dependencies (modern way)
uv pip install -e . --group dev

# Or using optional dependencies
uv pip install -e ".[dev]"

# Run tests
uv run pytest

# Format code  
uv run black src/
uv run isort src/

# Type checking
uv run mypy src/

# Run with specific Python version
uv run --python 3.11 python main.py

🔧 Build System & GPU Dependencies

This project uses modern Python packaging with GPU-first dependencies:

• 📦 Build Backend: hatchling - Fast, modern PEP 517/518 compliant builder
• 📋 Configuration: Pure pyproject.toml - No setup.py needed
• ⚡ Package Manager: uv - Ultra-fast Python package installer and resolver
• 🔄 Dependencies: Modern dependency-groups for development dependencies
• 🚀 GPU Dependencies: PyTorch with CUDA/MPS support, transformers with acceleration

Key Benefits:

• ✅ Zero setup.py - Pure pyproject.toml configuration
• ✅ Fast builds - Hatchling is significantly faster than setuptools
• ✅ Modern standards - Full PEP 517/518/621 compliance
• ✅ uv integration - Seamless uv run support without installation
• ✅ GPU-optimized - Automatic PyTorch CUDA/MPS dependencies
• ✅ Clean dependencies - No deprecated configurations

GPU Dependencies Included:

• PyTorch 2.8+ with CUDA 11.8+ / MPS support
• Transformers 4.55+ with GPU acceleration
• Apple Silicon: Native MPS backend
• NVIDIA: CUDA toolkit integration

CPU-Only Alternative:

If you need CPU-only inference, consider developing additional classes:

# Example CPU-optimized approach (not included in main project)
from ctransformers import AutoModelForCausalLM  # CPU-optimized
# This project focuses on GPU acceleration instead

pyproject.toml Structure:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[dependency-groups]
dev = ["pytest", "black", "isort", "mypy", "ruff"]

🏗️ Architecture

The application follows a clean, modular architecture:

src/
├── models/          # Data models and structures
├── utils/           # Utilities (config, logging, file handling, audio processing)  
├── asr/             # Automatic Speech Recognition modules
├── translation/     # Translation pipeline and models
├── subtitle/        # Subtitle processing and generation
└── ui/              # Tkinter GUI components

Core Components

🎯 Subtitle Generator (src/subtitle/subtitle_generator.py)

The main orchestrator that coordinates the entire subtitle generation pipeline:

• Video validation and metadata extraction
• Audio extraction and preprocessing
• ASR transcription with chunking for long videos
• Multi-language translation
• Subtitle formatting and export

🎤 ASR Module (src/asr/)

• Base ASR (base_asr.py): Abstract interface for ASR implementations
• Whisper ASR (whisper_asr.py): OpenAI Whisper integration with batching support

🌐 Translation Module (src/translation/)

• Translation Pipeline (translation_pipeline.py): Coordinates multi-stage translation
• HuggingFace Translator (huggingface_translator.py): Transformer-based GPU translation
• PyTorch Translator (pytorch_translator.py): GPU-first PyTorch implementation
• Multi-Stage Translator: Japanese → English → Traditional Chinese

Translation Backend Strategy:

• 🚀 Primary: HuggingFace Transformers with GPU acceleration (MPS/CUDA)
• 🌸 Advanced: PyTorchTranslator for SakuraLLM integration (specialized Japanese LLMs)
• ⚡ Alternative: Pure PyTorch with GPU optimization for any HuggingFace model
• ❌ Not Included: ctransformers (CPU-only focus conflicts with project goals)
• 🎯 Focus: Maximum GPU performance for real-time subtitle generation

🌸 PyTorchTranslator Features (SakuraLLM Ready):

• 🎯 LLM Support: Direct integration with large language models
• 🚀 GPU Optimization: MPS/CUDA acceleration with FP16 precision
• 🧠 Advanced Generation: Temperature, sampling, KV caching
• 📝 Prompt Engineering: Customizable prompts for specialized models
• 🎮 Memory Management: Efficient VRAM usage and model loading
• ⚖️ Model Agnostic: Works with any HuggingFace Transformers model

SakuraLLM Integration Example:

# Specialized Japanese light novel translation
sakura_translator = PyTorchTranslator(
    model_name="SakuraLLM/Sakura-1.5B-Qwen2.5-v1.0-GGUF",
    source_lang="ja", target_lang="zh",
    device="auto", torch_dtype="float16"
)

Note for CPU Users: This project prioritizes GPU acceleration. If you require CPU-only inference, consider implementing additional ctransformers-based classes, though this is outside the current project scope.

🖥️ UI Module (src/ui/)

• Main Window (main_window.py): Primary application interface
• Components: Settings dialog, preview dialog, progress tracking

🛠️ Utilities (src/utils/)

• Config Management (config.py): JSON-based configuration with defaults
• File Handler (file_handler.py): Video validation and file operations
• Audio Processor (audio_processor.py): Audio extraction and preprocessing
• Logger (logger.py): Structured logging with file rotation

⚙️ Configuration

The application uses a JSON configuration file with sensible defaults:

{
  "asr": {
    "model_name": "openai/whisper-large-v3",
    "device": "auto",
    "chunk_length": 30
  },
  "translation": {
    "ja_to_en_model": "Helsinki-NLP/opus-mt-ja-en", 
    "en_to_zh_model": "Helsinki-NLP/opus-mt-en-zh",
    "batch_size": 8
  },
  "dual_language": {
    "generate_both_languages": false,
    "japanese_suffix": "_ja",
    "translated_suffix": "_zh"
  },
  "ui": {
    "window_size": "800x600",
    "theme": "default"
  }
}

Configuration can be modified through:

• GUI Settings dialog (⚙️ Settings button)
• Direct JSON file editing (config.json)
• Command line arguments

🤖 Default Models & Requirements

🎙️ ASR Models (Speech Recognition)

Default: OpenAI Whisper Large-v3

• Model: openai/whisper-large-v3
• Size: ~3GB download
• Languages: 99+ languages (optimized for Japanese)
• Quality: Highest accuracy for Japanese speech recognition

Hardware Requirements:

Alternative ASR Models:

🌐 Translation Models

Japanese → English

• Model: Helsinki-NLP/opus-mt-ja-en
• Size: ~300MB download
• Specialty: Japanese to English translation
• Performance: Optimized for subtitle-length text

English → Chinese (Traditional)

• Model: Helsinki-NLP/opus-mt-en-zh
• Size: ~300MB download
• Specialty: English to Chinese translation
• Output: Traditional Chinese characters

Translation Hardware Requirements:

📊 Total System Requirements

Complete Setup (All Models):

Recommended Configurations:

🍎 Apple Silicon (Optimal):

• M2/M3/M4: 16GB unified memory
• Storage: 10GB free space
• Expected Performance: 8x realtime processing

🚀 NVIDIA GPU (Optimal):

• GPU: RTX 3070/4070 (8GB+ VRAM)
• RAM: 16GB system memory
• Expected Performance: 12x realtime processing

💻 Budget/Lower-end Hardware:

# Use smaller models in config.json
{
  "asr": {
    "model_name": "openai/whisper-medium"  # 1.5GB vs 3GB
  }
}

🔗 Model Links & Licenses

⚠️ Model Constraints & Limitations

Whisper Large-v3:

• Languages: Optimized for Japanese, may struggle with heavy dialect
• Audio Quality: Works best with clear speech, struggles with background noise
• Length: Optimal for 30-second chunks (longer may degrade accuracy)
• Real-time: Not suitable for live transcription (processing delay)

Translation Models:

• Domain: Trained on general text, may struggle with technical/domain-specific terms
• Length: Optimized for sentence-level translation (subtitle appropriate)
• Formality: May not preserve Japanese politeness levels in translation
• Cultural Context: Limited cultural nuance preservation

Hardware Constraints:

• Memory: All models must fit in VRAM/RAM simultaneously
• Processing: CPU fallback is 4-12x slower than GPU
• Storage: SSD recommended for faster model loading (vs HDD)

🎯 Hardware-Specific Setup Recommendations

🏆 High-End Setup (Recommended)

// config.json for optimal performance
{
  "asr": {
    "model_name": "openai/whisper-large-v3",
    "batch_size": 1,
    "device": "auto"
  },
  "translation": {
    "batch_size": 8,
    "device": "auto"
  }
}

Hardware: M3/M4 16GB+ or RTX 3070+ 8GB
Performance: 8-12x realtime
Use Case: Production subtitle generation

💰 Budget Setup (Balanced)

// config.json for balanced performance/memory
{
  "asr": {
    "model_name": "openai/whisper-medium",
    "batch_size": 1,
    "device": "auto"
  },
  "translation": {
    "batch_size": 4,
    "device": "auto"  
  }
}

Hardware: M1/M2 8GB or GTX 1660 6GB
Performance: 4-6x realtime
Use Case: Personal use, occasional processing

🔋 Minimal Setup (Emergency)

// config.json for low-resource systems
{
  "asr": {
    "model_name": "openai/whisper-small",
    "batch_size": 1,
    "device": "cpu"
  },
  "translation": {
    "batch_size": 1,
    "device": "cpu"
  }
}

Hardware: Any system with 8GB RAM
Performance: 1-2x realtime
Use Case: Testing, very low-end hardware

📥 First-Time Setup & Downloads

Automatic Model Downloads:

# First run will download all models (~3.6GB)
uv run python main.py

# Downloads will occur to:
# ~/.cache/huggingface/transformers/
# Total download time: 5-15 minutes (depending on internet)
# Note: Models cache to ~/.cache/huggingface/ and won't re-download

⚡ Cache Optimization:

Models are cached to ~/.cache/huggingface/ and reused across runs. However, you might see downloads for:

• Format updates: safetensors (preferred) vs pytorch_model.bin
• Missing components: tokenizer files, config updates

Cache Status Check:

# Check cached models
ls ~/.cache/huggingface/hub/

# Cache size
du -sh ~/.cache/huggingface/

Force Pre-download:

# Pre-download all model formats to avoid future downloads
uv run python predownload_models.py

Pre-download Models (Optional):

# Pre-download script (optional)
uv run python -c "
from transformers import AutoModel, AutoTokenizer, pipeline

# Download ASR model
pipeline('automatic-speech-recognition', model='openai/whisper-large-v3')

# Download translation models  
pipeline('translation', model='Helsinki-NLP/opus-mt-ja-en')
pipeline('translation', model='Helsinki-NLP/opus-mt-en-zh')

print('All models downloaded successfully!')
"

🌸 SakuraLLM Integration (Advanced Option)

🎯 Alternative: SakuraLLM for Superior Japanese Translation

For users seeking highest quality Japanese→Chinese translation, we support SakuraLLM integration via the PyTorchTranslator:

📋 SakuraLLM Model Options:

✅ Recommended: Sakura-1.5B-Qwen2.5-v1.0

• 🎯 Optimal Balance: Quality vs Hardware Requirements
• 🚀 Real-time Ready: Fast enough for subtitle generation
• 💾 Accessible: Fits on M1 8GB and GTX 1660 6GB
• 🎨 Specialized: Light novel translation optimized

⚙️ SakuraLLM Integration:

# config.json - SakuraLLM setup
{
  "translation": {
    "use_sakura": true,
    "sakura_model": "SakuraLLM/Sakura-1.5B-Qwen2.5-v1.0-GGUF",
    "source_lang": "ja",
    "target_lang": "zh", 
    "device": "auto",
    "torch_dtype": "float16"
  }
}

# Usage with PyTorchTranslator
from src.translation.pytorch_translator import PyTorchTranslator

sakura = PyTorchTranslator(
    model_name="SakuraLLM/Sakura-1.5B-Qwen2.5-v1.0-GGUF",
    source_lang="ja",
    target_lang="zh",
    device="auto",  # GPU-first
    force_gpu=True
)

🌸 SakuraLLM Advantages:

• 📚 Light Novel Optimized: Specialized for Japanese fiction/anime content
• 👥 Character Context: Better character name and pronoun handling
• 🎭 Cultural Nuance: Superior cultural context preservation
• 📖 Terminology Support: Built-in glossary/dictionary support
• 🎯 Subtitle Appropriate: Optimized for short text segments

📖 Custom Terminology Glossary:

SakuraLLM supports custom terminology dictionaries for consistent translation of specialized terms:

# prompts/my_glossary.txt
# Format: 原文->译文 (Japanese->Chinese)
お兄ちゃん->哥哥
先輩->前輩
魔法少女->魔法少女

Usage in config.json:

{
  "sakura": {
    "prompt_template": "my_glossary"
  }
}

The glossary ensures consistent translation of character names, honorifics, and domain-specific terminology.

⚠️ SakuraLLM Limitations:

• 🌐 Language Pair: Japanese→Chinese only (no English intermediate)
• 📖 Domain: Optimized for light novels, may struggle with technical content
• 💾 Size: Larger than Helsinki-NLP models (2-4GB vs 300MB)
• ⚖️ License: CC-BY-NC-SA-4.0 (non-commercial)
• 🔧 Complexity: Requires more sophisticated prompt engineering

🎯 When to Use SakuraLLM:

• ✅ Anime/Light Novel content translation
• ✅ High-quality Japanese→Chinese required
• ✅ Have 8GB+ VRAM/RAM available
• ✅ Non-commercial use acceptable

🎯 When to Use Default (Helsinki-NLP):

• ✅ General purpose translation
• ✅ Resource-constrained systems
• ✅ Commercial use required
• ✅ Multi-language pairs needed

⚡ Performance Comparison

Translation Backend Comparison:

GPU vs CPU Performance (Approximate benchmarks):

Why GPU-First Design:

• 🚀 Speed: 4-12x faster processing than CPU-only solutions
• 🧠 Efficiency: Better memory bandwidth utilization
• ⚡ Real-time: Enables real-time subtitle generation
• 🔋 Power: More energy-efficient on Apple Silicon

CPU-Only Alternative Guidance:

If you're limited to CPU-only inference:

1. Performance: Expect 4-12x slower processing
2. Implementation: Consider ctransformers with GGUF models
3. Project Scope: CPU optimization is outside current focus
4. Recommendation: Use smaller models (whisper-small, smaller translation models)

📋 Workflow

1. 🎬 Video Input: Select video file through file browser
2. ⚙️ Configuration: Choose target languages, dual language output, and settings
3. 🔍 Validation: Verify video format and extract metadata
4. 🎵 Audio Extraction: Extract audio using FFmpeg
5. 🎙️ ASR Processing: Generate Japanese transcription with Whisper
6. 🌐 Translation: Translate to English and Traditional Chinese
7. 📝 Subtitle Generation: Create formatted subtitle files (single or dual language)
8. 👁️ Preview & Export: Review results and export SRT files

🚦 Error Handling

• Graceful Degradation: Continues processing if non-critical steps fail
• User Feedback: Clear error messages with actionable suggestions
• Resource Management: Automatic cleanup of temporary files and GPU memory
• Logging: Comprehensive logging for debugging

🎛️ Advanced Features

Chunked Processing

• Automatically splits long videos into chunks for memory efficiency
• Overlapping chunks prevent word boundary issues
• Configurable chunk size and overlap

Device Management

• Automatic GPU detection and fallback to CPU
• Memory-conscious model loading/unloading
• Configurable device preferences

Subtitle Optimization

• Automatic text cleaning and formatting
• Optimal line breaking for readability
• Configurable timing constraints

📊 Performance Considerations

• Memory Usage: Models are loaded/unloaded as needed
• GPU Support: CUDA acceleration when available
• Batch Processing: Efficient batching for translations
• Chunking: Handles videos of any length

🔮 Future Enhancements

• CLI mode implementation
• Additional subtitle formats (VTT, ASS)
• Speaker diarization
• Custom model fine-tuning
• Web interface
• Docker deployment
• Additional language pairs

📚 Documentation

📋 Project Documentation

• Progress Summary - Complete overview of recent improvements and transformations
• Technical Changelog - Detailed technical implementation documentation
• UV Tool Strategy - Why we use uv tool for development tools instead of project dependencies

🚀 Recent Major Updates

• Native Whisper Implementation - Refactored from pipeline to native generate() method
• Zero Experimental Warnings - Eliminated all ASR warnings for clean operation
• Production Ready - Following Whisper paper best practices (Section 3.8)
• Performance Optimized - Better memory usage and generation efficiency
• Dual Language Subtitle Generation - Support for simultaneous Japanese and translated subtitle file output

🎯 Key Improvements

• ✅ No Token Limits - Removed artificial generation constraints
• ✅ Silent Operation - Clean logs without experimental warnings
• ✅ Better Quality - Using Whisper's intended architecture
• ✅ Full Device Support - Maintained MPS/CUDA/CPU compatibility
• ✅ Dual Language Output - Configurable Japanese + translated subtitle file generation

For complete technical details, see TECHNICAL_CHANGELOG.md

🤝 Contributing

The codebase is designed for extensibility:

• New ASR Models: Extend BaseASR class
• New Translation Models: Extend BaseTranslator class
• New Export Formats: Add methods to SubtitleFile class
• New UI Components: Add to src/ui/components/

📄 License

[License information]

🙏 Acknowledgments

• OpenAI for Whisper ASR models and the foundational research
• Whisper Paper (Section 3.8) - "Robust Speech Recognition via Large-Scale Weak Supervision" for implementation methodology and best practices
• SakuraLLM Project for specialized Japanese-to-Chinese translation models optimized for light novel and anime content
• Hugging Face for transformer models and pipeline APIs
• Helsinki-NLP for OPUS-MT translation models
• FFmpeg Project for comprehensive audio processing capabilities

📚 Documentation

Comprehensive Guides:

• Hardware Acceleration Guide - Cross-platform acceleration setup and performance optimization
• Complete Progress Summary - Detailed development progress and achievements
• Apple Silicon Setup Guide - M1/M2/M3/M4 optimization instructions
• SakuraLLM Integration Guide - SakuraLLM model setup and configuration
• UV Tool Strategy - Clean development environment management

Technical Documentation:

• Source Structure - Codebase architecture and module organization
• Makefile Improvements - Development workflow automation
• Optimization Summary - Performance enhancements and code quality improvements

---

Built with ❤️ for the subtitle generation community 🌸