NanoI2V: Building an Image-to-Video Model from Scratch

Overview

NanoI2V is a from-scratch implementation of an Image-to-Video (I2V) generation pipeline.

The project focuses on understanding and implementing the core concepts and building blocks behind modern video generation systems such as:

• Variational Autoencoders (VAE)
• Latent Video Modeling
• Diffusion / Flow Matching
• DiT Transformers
• Cross-Attention Conditioning
• Classifier-Free-Guidance

---

The series is published as a dedicated website with structured lessons, explanations, and code walkthroughs: shubham2376g.github.io/NanoI2V

Each topic is a self-contained lesson - read in order or jump to what you need.

---

What This Series Covers

This series explores the core building blocks behind modern image-to-video (I2V) models, including topics such as:

Additional topics and modules will be added as the series evolves.

---

Repository Structure

NanoI2V/
├── vae/
│   ├── conv.py                 # CausalConv3D implementation
│   ├── blocks.py               # 3D ResBlocks and Spatial Attention modules
│   ├── encoder.py              # VAE encoder
│   ├── decoder.py              # VAE decoder
│   └── vae.py                  # VAE model definition
│
├── dit/
│   ├── rope.py                 # 3D Rotary Positional Embeddings (RoPE)
│   ├── attention.py            # Self-Attention and Cross-Attention layers
│   ├── blocks.py               # DiT blocks with Adaptive LayerNorm (adaLN)
│   └── dit.py                  # Diffusion Transformer (DiT) architecture
│
├── flow/
│   └── scheduler.py            # Flow Matching scheduler
│
├── conditioning/
│   └── encoders.py             # Text and image conditioning encoders
│
├── data/
│   ├── download_vidgen.py      # Dataset download utilities
│   ├── prepare_vidgen.py       # Dataset preparation pipeline
│   └── preprocess_vae.py       # VAE preprocessing scripts
│
├── docs/
│   └── index.html              # Project website (GitHub Pages)
│
├── train_vae.py                # VAE training script
├── train_dit.py                # DiT training script
├── inference_dit.py            # Inference and video generation
│
└── README.md                   # Project documentation

---

High-Level DiT Pipeline

NanoI2V follows the latent diffusion paradigm, where the Diffusion Transformer (DiT) operates entirely in the latent space produced by the VAE instead of directly on pixels.

During each denoising step, the model takes:

• Noisy video latents to be denoised
• Image latents extracted from the conditioning image
• Text tokens from the prompt encoder
• Image tokens from the image encoder
• Timestep embeddings indicating the current diffusion step

These inputs are processed by a stack of DiT blocks that iteratively refine the latent representation using attention and conditioning mechanisms. The final layer predicts the velocity field used by the flow-matching solver to progressively generate the target video latent.

Simplified overview of the NanoI2V generation pipeline. The figure emphasizes the flow of information between components rather than the exact implementation. Detailed architectural components are introduced in later chapters.

---

Results

The following results were generated using the models implemented in this repository.

VAE Reconstruction Results

The VAE is trained to compress video clips into a latent representation and reconstruct them with minimal quality loss.

---

DiT Video Generation Results

The Diffusion Transformer (DiT) is trained in latent space and generates video sequences conditioned on an input image.

Example 1

---

Example 2

---

Prerequisites

A basic understanding of the following will help:

• PyTorch basics
• Transformer architecture and attention mechanisms
• LLM fundamentals
• Diffusion model intuition (helpful but not required)

If you've worked with LLMs before, many concepts here will feel familiar.

---

🚀 Google Colab Smoke Run

Before downloading a large dataset or launching full training, you can verify that your environment is set up correctly by running the smoke tests included in examples/.

These tests use tiny model configurations and synthetic data, allowing you to validate the complete NanoI2V pipeline in just a few minutes on a free Google Colab GPU.

1. Clone the repository

git clone https://github.com/Shubham2376G/NanoI2V.git
cd NanoI2V

2. Install NanoI2V

Install the project in editable mode:

pip install -e .

This makes the local vae, dit, flow, and other project modules importable so that all smoke tests can run correctly.

3. (Optional) Generate a dummy dataset

To verify the data loading pipeline without downloading a real dataset, generate a tiny synthetic dataset:

python examples/make_smoke_data.py

This creates a small dataset under data/smoke/ containing randomly generated videos and captions.

---

Available Smoke Tests

All smoke tests are located in the examples/ directory.

Run any smoke test with

python examples/<script_name>.py

For example,

python examples/smoke_train_vae.py
python examples/smoke_video_dit.py

If all smoke tests complete successfully, your NanoI2V installation is ready for training on a real dataset.

---

⭐ Support the Project

If you find this useful:

• Star the repository ⭐
• Share it with others interested in diffusion or video models