🎬 OpenSora Multi-GPU Inference Pipelines

A collection of production-ready, multi-GPU inference pipelines for video generation, image-to-video conversion, and depth estimation using Hugging Face models.

📦 Project Structure

opensora/
├── text_to_video/              # Text-to-video generation
│   ├── simple_inference.py
│   ├── multi_gpu_inference.py
│   ├── batch_inference.py
│   ├── compare_models.py
│   └── README.md
│
├── image_to_video/             # Image-to-video conversion
│   ├── simple_image_to_video.py
│   ├── image_to_video_multi_gpu.py
│   ├── batch_image_to_video.py
│   └── README.md
│
└── depth_estimation/           # Depth estimation
    ├── simple_depth_estimation.py
    ├── multi_gpu_depth_estimation.py
    ├── batch_depth_estimation.py
    ├── compare_models.py
    └── README.md

🚀 Features

Common Features Across All Pipelines

• ✅ Multi-GPU Support - Automatic distribution across available GPUs
• ✅ Memory Optimization - Smart memory management to prevent OOM errors
• ✅ Multiple Models - Support for various state-of-the-art models
• ✅ Batch Processing - Efficient processing of multiple inputs
• ✅ Jupyter/Colab Ready - Works seamlessly in notebooks
• ✅ Comprehensive Documentation - Detailed guides and examples

Text-to-Video Pipeline

• Generate videos from text descriptions
• Support for ModelScope, Zeroscope, and more
• Customizable resolution, duration, and guidance

Image-to-Video Pipeline

• Convert static images to videos
• Text-guided animation
• Support for Stable Video Diffusion and I2VGen-XL
• Motion control and quality settings

Depth Estimation Pipeline

• Monocular depth estimation from images
• 7 pre-configured models (98MB to 1.4GB)
• Colorized visualization with custom colormaps
• Raw depth arrays for 3D applications

📥 Installation

Install All Dependencies

# Clone or download the project
cd opensora

# Install dependencies for all pipelines
pip install torch>=2.0.0 transformers>=4.35.0 diffusers>=0.25.0 accelerate>=0.24.0 \
            pillow>=10.0.0 matplotlib>=3.7.0 "numpy<2.0.0" opencv-python>=4.8.0

Or Install Per Pipeline

# Text-to-video only
pip install -r text_to_video/requirements.txt

# Image-to-video only
pip install -r image_to_video/requirements.txt

# Depth estimation only
pip install -r depth_estimation/requirements.txt

Jupyter/Colab

!pip install -q torch transformers diffusers accelerate pillow matplotlib "numpy<2.0.0"

🎯 Quick Start Examples

1. Text-to-Video

from text_to_video.simple_inference import simple_text_to_video

# Generate video from text
simple_text_to_video(
    prompt="A cat playing piano",
    output_path="cat_piano.mp4"
)

2. Image-to-Video

from image_to_video.simple_image_to_video import simple_image_to_video

# Convert image to video
simple_image_to_video(
    image_path="photo.jpg",
    output_path="animated.mp4"
)

3. Depth Estimation

from depth_estimation.simple_depth_estimation import simple_depth_estimation

# Estimate depth from image
simple_depth_estimation(
    image_path="scene.jpg",
    output_path="depth_map.png"
)

🖥️ Multi-GPU Usage

All pipelines support multi-GPU automatically:

# Text-to-video with multi-GPU
from text_to_video.multi_gpu_inference import generate_video_multi_gpu

generate_video_multi_gpu(
    prompt="Astronaut riding a horse",
    use_multi_gpu=True  # Automatically distributes across GPUs
)

# Image-to-video with multi-GPU
from image_to_video.image_to_video_multi_gpu import image_to_video_multi_gpu

image_to_video_multi_gpu(
    image_path="input.jpg",
    use_multi_gpu=True
)

# Depth estimation with multi-GPU
from depth_estimation.multi_gpu_depth_estimation import estimate_depth_multi_gpu

estimate_depth_multi_gpu(
    image_path="input.jpg",
    use_multi_gpu=True
)

📊 Performance

Tested on 2x Tesla T4 GPUs (14.7GB each):

🎨 Model Support

Text-to-Video Models

• ModelScope (damo-vilab/text-to-video-ms-1.7b)
• Zeroscope (cerspense/zeroscope_v2_576w, zeroscope_v2_XL)
• AnimateDiff
• And more...

Image-to-Video Models

• Stable Video Diffusion (stabilityai/stable-video-diffusion-img2vid)
• I2VGen-XL (ali-vilab/i2vgen-xl)

Depth Estimation Models

• DPT (Intel/dpt-large, Intel/dpt-hybrid-midas)
• Depth Anything V2 (depth-anything/Depth-Anything-V2-*)
• GLPN (vinvino02/glpn-kitti, vinvino02/glpn-nyu)

📚 Documentation

Each pipeline has comprehensive documentation:

• README.md - Full documentation with examples
• QUICKSTART.md - Get started in 30 seconds
• MULTI_GPU_GUIDE.md - Multi-GPU optimization
• TROUBLESHOOTING.md - Common issues and solutions

🔧 Configuration

GPU Memory Optimization

import os
import torch
import gc

# Set memory optimization
os.environ['PYTORCH_CUDA_ALLOC_CONF'] = 'expandable_segments:True'

# Clear GPU cache
torch.cuda.empty_cache()
gc.collect()

Select Specific GPUs

# Use only GPU 0 and 1
export CUDA_VISIBLE_DEVICES=0,1

# Or in Python
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0,1'

🎯 Use Cases

Content Creation

• Generate promotional videos from text
• Animate product photos
• Create depth effects for images

Research & Development

• Video generation experiments
• Depth estimation for 3D reconstruction
• Animation research

Production Applications

• Batch video generation
• Automated content pipeline
• Large-scale image processing

💡 Best Practices

1. Start Small

# Test with small model first
model_id="depth-anything-small"
max_image_size=512

2. Use Batch Processing

# Much faster than processing individually
python batch_depth_estimation.py --input-dir ./images

3. Monitor Memory

import torch

# Check GPU memory
for i in range(torch.cuda.device_count()):
    print(f"GPU {i}: {torch.cuda.memory_allocated(i)/1e9:.2f}GB")

4. Clear Memory Between Runs

torch.cuda.empty_cache()
import gc; gc.collect()

🐛 Troubleshooting

Common Issues

CUDA Out of Memory:

• Reduce image/video size
• Use smaller models
• Enable multi-GPU
• Clear GPU cache

Slow Performance:

• Check GPU is being used: torch.cuda.is_available()
• Use smaller models
• Reduce resolution
• Enable multi-GPU

Model Download Fails:

• Check internet connection
• Set cache directory: export HF_HOME=/path/to/cache
• Try different model

See individual TROUBLESHOOTING.md files for detailed solutions.

🧪 Testing

Test each pipeline:

# Test text-to-video
python text_to_video/test_installation.py

# Test image-to-video
python image_to_video/simple_image_to_video.py --help

# Test depth estimation
python depth_estimation/test_installation.py

📦 System Requirements

Minimum

• Python 3.8+
• 8GB RAM
• CPU (slow but works)

Recommended

• Python 3.10+
• 16GB RAM
• NVIDIA GPU with 8GB+ VRAM
• CUDA 11.8+

Optimal

• Python 3.11
• 32GB RAM
• Multiple NVIDIA GPUs with 12GB+ VRAM each
• CUDA 12.0+

🤝 Contributing

Contributions welcome! Each pipeline is self-contained and follows the same structure for easy extension.

📄 License

Individual models have their own licenses. Check model cards on Hugging Face.

🔗 Resources

• Hugging Face Transformers
• Diffusers Documentation
• PyTorch Multi-GPU

📧 Support

For issues:

1. Check the pipeline-specific TROUBLESHOOTING.md
2. Run test_installation.py
3. Review examples in QUICKSTART.md
4. Check GPU memory with nvidia-smi

🎓 Examples

See the examples.py file in each pipeline directory for 10+ ready-to-run examples covering:

• Simple usage
• Multi-GPU optimization
• Batch processing
• Model comparison
• Custom configurations
• Memory-efficient modes
• And more!

---

Ready to start? Choose a pipeline and check its QUICKSTART.md file!

# Text-to-video
cd text_to_video && cat QUICKSTART.md

# Image-to-video
cd image_to_video && cat QUICKSTART.md

# Depth estimation
cd depth_estimation && cat QUICKSTART.md