ZipRight Virtual Try-On (VTO) Research & Benchmark Repository

Welcome to the ZipRight Virtual Try-On (VTO) research repository, containing a modular 15-stage pipeline, benchmark suites, and optimization scripts designed for edge deployments.

---

1. Project Overview & Architecture

ZipRight VTO is a production-functional, modular try-on system designed to process human clothing try-on tasks by splitting the problem into 15 specific, isolated stages. This allows fine-grained optimizations at each layer (e.g. separate networks for parsing, geometry estimation, occlusion handling, shadow matching, and super-resolution) rather than relying on a single end-to-end black box model.

15-Stage VTON Pipeline Architecture Diagram

graph TD
    A[Inputs: Person Image & Garment Image] --> Stage1[Stage 1: Human Parsing ONNX/OpenPose]
    Stage1 --> Stage2[Stage 2: Body Geometry DensePose]
    Stage1 --> Stage3[Stage 3: Garment Understanding OCR]
    Stage1 --> Stage6[Stage 6: Occlusion Reasoning]
    Stage1 --> Stage7[Stage 7: Identity preservation Face Lock]
    
    Stage3 --> Stage4[Stage 4: Garment Flow Warping]
    Stage4 --> Stage5[Stage 5: Physics & Shading Map]
    
    Stage5 --> Stage8[Stage 8: Core Diffusion Generator Leffa]
    Stage2 --> Stage8
    Stage6 --> Stage8
    Stage7 --> Stage8
    
    Stage8 --> Stage9[Stage 9: Logo Preservation]
    Stage9 --> Stage10[Stage 10: Fabric Texture Enhancement]
    Stage10 --> Stage11[Stage 11: Collar Reconstruction]
    Stage11 --> Stage12[Stage 12: Sleeve Reconstruction]
    Stage12 --> Stage13[Stage 13: Lighting Adaptation]
    Stage13 --> Stage14[Stage 14: Super Resolution 2K]
    Stage14 --> Stage15[Stage 15: Quality Verification & SSIM]
    Stage15 --> Out[Output: result.png]

---

2. The 15-Stage VTO Pipeline Details

1. Stage 1: Human Parsing - Generates semantic segmentations of body parts (arms, torso, neck, legs) and joint keypoints.
2. Stage 2: Body Geometry Estimation - DensePose mapping to project a 3D coordinate model onto the body.
3. Stage 3: Garment Understanding - OCR and layout analysis to extract sleeve length, collar type, and text features.
4. Stage 4: Garment Flow Estimation - Thin-Plate Spline (TPS) or dense flow mapping warping the garment onto the body geometry.
5. Stage 5: Garment Physics & Folding - Physics-informed wrinkle generation and shaded map blending.
6. Stage 6: Occlusion Reasoning - Detects overlapping layers like hands, bags, or cross-body occlusions.
7. Stage 7: Identity Embedding Extraction - Extracts facial features and creates hair/face safety preservation masks.
8. Stage 8: Latent Diffusion Denoising - Generates the high-fidelity try-on composition using the Leffa backbone.
9. Stage 9: Logo Attention Alignment - Blends target brand logos and labels back over the warped garment.
10. Stage 10: Fabric Texture Enhancement - Post-denoising high-frequency texture injection.
11. Stage 11: Collar Structural Reconstruction - Fine-tunes structural neckline/collar edges.
12. Stage 12: Sleeve Edge Reconstruction - Cleans sleeve borders and arm/clothing boundary intersections.
13. Stage 13: Lighting & Shadow Adaptation - Harmonizes lighting directions, shadows, and environment contrast.
14. Stage 14: Real-ESRGAN Sharpening - Upscales the generated output from base 768x1024 to 2K resolution (1536x2048).
15. Stage 15: Quality & Similarity Verification - Computes SSIM metrics for face preservation and garment fidelity.

---

3. Folder Structure

zipright-vto-lab/
├── engines/                      # Core VTO engines
│   ├── base.py                   # Base class definitions
│   ├── catvton_engine.py         # CatVTON model wrapper
│   ├── idmvton_engine.py         # IDM-VTON model wrapper
│   ├── leffa_engine.py           # Leffa wrapper with dynamic VRAM paging
│   ├── zipright.py               # Main 15-Stage pipeline orchestrator
│   └── zipright_stages/          # Implementation files for each stage
├── inputs/                       # Sample inputs
├── models/                       # Submodule repositories
│   ├── catvton/
│   ├── comfyui_controlnet_aux/
│   └── idmvton/
├── scripts/                      # Execution and benchmark scripts
│   ├── benchmark_parsing.py
│   ├── run_all.py
│   └── run_zipright.py
├── .gitignore
├── README.md
└── README_MODELS.md              # Weight download instructions

---

4. Installation & Dependencies

Ensure you have a modern GPU with CUDA support (e.g. RTX 3050 Laptop or higher).

# Clone the repository
git clone <repository_url>
cd zipright-vto-lab

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install requirements
pip install -r requirements.txt

Note: For downloading the required AI model weight checkpoints, please follow the steps in README_MODELS.md.

---

5. Usage & Benchmark Instructions

How to Run the Pipeline

To run the full 15-stage pipeline on sample input:

python scripts/run_zipright.py

Outputs are written to outputs/zipright/result.png and debugging intermediate maps are saved in outputs/debug/generator/.

How to Run Benchmarks

To run the complete engine benchmark comparison (comparing ZipRight against baseline CatVTON and IDM-VTON):

python scripts/run_all.py

---

6. Memory Optimization Strategy (RTX 3050 6GB)

Running state-of-the-art VTO models typically requires > 12GB of VRAM. To run ZipRight comfortably under a strict 6GB VRAM limit (such as on RTX 3050 Laptop GPUs), the following memory techniques are implemented in engines/leffa_engine.py:

• Sequential Model Swapping / Offloading: We isolate submodels (VAE, Reference UNet, Generative UNet) and load only one model to the GPU at a time.
• Immediate CPU-side Cache Offloads: Models are transferred to CPU (.to("cpu")) and deleted from the GPU immediately after finishing their step.
• Aggressive Cache Cleans: Calls to torch.cuda.empty_cache() and Python garbage collection gc.collect() are forced between each phase.
• Half Precision (FP16): Inference is executed in dynamic autocast FP16 contexts, cutting model size and memory bandwidth requirements in half.

---

7. Current Implementation Status

---

8. Known Limitations & Future Roadmap

Limitations

• GPU-to-CPU Paging Overhead: The sequential offloading strategy introduces CPU-GPU communication overhead, increasing pipeline execution latency (approx 20–30s per image).
• Garment SSIM Tuning: Post-processing stages are baseline implementations and require tuning to optimize SSIM scores further.

Future Roadmap

• Real-time GPU model compilation: Integrate PyTorch 2.x torch.compile or TensorRT to accelerate inference steps.
• Asymmetrical Pipeline Execution: Parallelize Stage 1–7 estimations while preparing latents to optimize total end-to-end VTO run times.