ViT to DiT Vision Coursework

This repo is a runnable study track for learning modern transformer-based vision models from first principles through current diffusion-transformer video generation systems. It is built for the train-cu124 conda environment and the 4x H200 reservation pod available in this workspace.

The path is deliberate:

1. ResNet -> ViT: understand patchification, CLS tokens, positional embeddings, and why pure attention needed large-scale pretraining.
2. ViT -> DeiT -> Swin: study the fixes for data hunger, inductive bias, and dense-vision scalability.
3. ViT -> DINO / MAE: study self-supervised representation learning and why masked prediction works especially well for vision transformers.
4. U-Net diffusion -> LDM -> DiT: understand the generative pivot from convolution-heavy denoisers to latent transformers.
5. VideoMAE / ViViT -> Video Diffusion -> STDiT / CogVideoX: extend the image story into spatiotemporal transformers and video generation.

Quick Start

Activate the environment and install the missing packages:

source /mnt/ddn/prod-runs/jeonghunpark/envs/miniforge3/bin/activate train-cu124
bash scripts/bootstrap_env.sh

Prepare the datasets used by the labs:

python scripts/prepare_data.py --dataset tiny-imagenet
python scripts/prepare_data.py --dataset imagenette-160
python scripts/prepare_data.py --dataset cifar10
python scripts/prepare_data.py --dataset moving-mnist
python scripts/prepare_data.py --dataset ucf101-subset
python scripts/prepare_data.py --dataset webvid-mini

Run a module:

python scripts/run_module.py --module 01_vit --config labs/01_vit/configs/tiny_imagenet_vit.yaml
python scripts/run_module.py --module 02_deit --config labs/02_deit/configs/tiny_imagenet_distill.yaml
python scripts/run_module.py --module 04_ssl_vits --config labs/04_ssl_vits/configs/mae_tiny_imagenet.yaml --stage pretrain
python scripts/run_module.py --module 04_ssl_vits --config labs/04_ssl_vits/configs/mae_tiny_imagenet.yaml --stage finetune
python scripts/run_module.py --module 08_video_dit --config labs/08_video_dit/configs/cogvideox_lora.yaml --stage lora

Evaluate a module:

python scripts/eval_module.py --module 01_vit --config labs/01_vit/configs/tiny_imagenet_vit.yaml
python scripts/eval_module.py --module 08_video_dit --config labs/08_video_dit/configs/moving_mnist.yaml --stage tiny

Run the local smoke tests:

python scripts/smoke_test.py

Module Map

Directory Layout

.
├── labs/                  # Per-module code, configs, and instructions
├── notes/                 # Literature review and transition notes
├── scripts/               # Shared bootstrap, data prep, run/eval tooling
├── syllabus/              # Week-by-week course structure
├── third_party/           # Shallow clones of upstream repos
├── artifacts/             # Checkpoints, metrics, generated outputs
└── data/                  # Prepared datasets (created on first use)

Data And Storage Defaults

• VLA_STUDY_DATA_ROOT defaults to <repo>/data.
• VLA_STUDY_ARTIFACT_ROOT defaults to <repo>/artifacts.
• VLA_STUDY_CACHE_ROOT defaults to /mnt/nvme/${USER}/vla-study-cache if /mnt/nvme exists, otherwise <repo>/.cache.

This keeps durable assets on DDN and scratch/cache activity on NVMe.

External Repos

The coursework uses the following upstream repos under third_party/:

• vision_transformer
• deit
• Swin-Transformer
• mae
• DiT
• diffusers
• CogVideo
• Open-Sora

They are not edited by the local coursework wrappers.

Primary Papers

See notes/papers.md for the full list and notes/evolution.md for the “limitation -> next approach” transitions.