PyTorchTutorial

This repository contains my snippets and sample codes for developing deep learning application with Pytorch.

Installation

conda create -n PyTorchTutorial python=3.12 -y
. "$HOME/anaconda3/etc/profile.d/conda.sh"
conda activate PyTorchTutorial

Install PyTorch

For a machine with a working NVIDIA driver and nvidia-smi, install the current CUDA build from the official PyTorch wheel index:

python -m pip install --upgrade pip
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126

pip install jupyterlab matplotlib seaborn scikit-learn pydot torchviz mlflow timm opencv-python albumentations tqdm tensorboard wandb kagglehub pytorch-lightning shap

monai[all] is intentionally not installed by default because it is large and pulls many optional dependencies. Install it only if you need the medical imaging notebooks:

pip install "monai[all]"

Optional system packages

If you want to view generated graphviz .dot files:

sudo apt-get install graphviz
sudo apt-get install xdot

Repo location

If you want the env-local src symlink that points to this repo:

ln -s /home/$USER/workspace/PyTorchTutorial /home/$USER/anaconda3/envs/PyTorchTutorial/src

Updating packages

Do not use conda update --all for this env. It tends to churn the full stack and is a common way to break a working PyTorch/CUDA setup.

Upgrade only what you actually need, for example:

pip install --upgrade torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126

PyTorch Fundamentals

• PyTorch Tensor Basics & Data Types
• Einsum Operator
• Grad Package
  • Computational Graph
  • Autograd
  • Dynamic Computational Graph
  • Detach
  • Exclusion from the DAG
  • Leaf Tensor
  • No Grad
  • Zero Grad
  • requires_grad
• Model Saving & Loading (Serialization)
• Parameters Registration, ModuleList

Neural Network Basics

• Neural Networks, Manifolds, and Topology
• Back Propagation
• Activation Functions
• Loss Functions
• Inductive Bias

Training Process

• Optimizer Package
• Learning Rate & Learning Rate Scheduler Schedulers
• Regularization
• Dropout Layers
• Normalization
  • Batch Normalization
  • Layer Normalization
  • Instance Normalization
  • Group Normalization
• Weight Initialization Strategies
• Evaluation vs Training Mode (Learning Monitoring)
  • Training, Validation, and Test Set
  • Monitor for Overfitting
  • Early Stopping
  • Visualize Metrics
• Real World Practices for Training and Regularization and PyTorch training template
• Function Approximation

---

CNN Building Blocks

• Convolution, Cross-Correlation, Transposed Convolution (Deconvolution), 1x1 Convolution
• Shape of Output
• RGB Image Convolution
• Convolution as Matrix Multiplication
• Conv2d class vs conv2d function
• Unfold/ fold
• Padding, Stride, Dilation
• Pooling (Max, Average, Adaptive), Order of Relu and Max Pooling
• Feature Map
• Convolution is translation-Equivariant, not Translation-Invariant
• Visualize Architecture of Neural Network

---

Modern Vision Architectures

CNN Architectures

• VGG
• ResNet
• RegNet
• EfficientNet
• MobileNet
• ConvNeXt
• DenseNet
• Inception

---

Image Segmentation

• Semantic, Instance, and Panoptic Segmentation -Panoptic Architectures
• U-Net
• DeepLab
• SAM 3
• Saliency Detection

---

Object Detection

• Object Detection Evaluation Metrics
• YOLO (You Only Look Once)
• Faster R-CNN
• SSD (Single Shot Detector)
• RetinaNet
• DETR
• Mask R-CNN - Instance segmentation + detection

---

Medical Imaging

• MONAI (Medical Open Network for AI)
• nnU-Net (Biomedical Image Segmentation)

---

Image Preprocessing & Augmentation Workflows

• DataLoader, Custom Dataset, TensorDataset, ImageFolder, random_split, Subset
• Transforms, Pre-Processing
• Data Augmentation
• OpenCV and PIL Image Format

---

Attention & Transformers

• Transformer Architecture
• Relative Positional Encoding
• Vision Transformer
• Swin Transformer
• DINO
• CLIP
• DeiT
• Pyramid Vision Transformer
• Feature Pyramid Network (FPN)
• Temporal Transformer

---

Foundation Models

General-purpose, self-supervised backbones that are frozen and reused across many downstream tasks (classification, retrieval, segmentation, depth, correspondence) via lightweight heads.

• DINOv3 — one backbone, many outputs
  • The three outputs: CLS, patch, register tokens
  • Global embedding → retrieval / classification / clustering
  • Patch features → segmentation, depth, correspondence
  • Depth estimation script (CLI)

---

Advanced Topics & Research Trends

• Encoder/ Decoder Architecture
• Variational Autoencoders
• Diffusion Models (Denoising Score Matching)
• Contrastive Learning
• Zero-shot & Few-shot Learning
• Transfer learning, Fine tuning, Backbone, Neck, Head
• Ensembling Models
• Flow Matching
• Making Network Deterministic
• Knowledge Distillation
• Neural Architecture Search (NAS), and Design Spaces
• PyTorch Image Models ( timm )
• Stochastic Depth
• 3D CNN
• Squeeze-and-Excitation Networks (SENet)
• Convolutional Block Attention Module (CBAM)
• MBConv
• Fused-MBConv
• Double Descent
• Receptive Field
• Universal Approximation Theorem
• Degrading Problem in Deep Learning
• PyTorch Lightning
• Gradient Clipping

---

Multimodal Models

• Vision-Language Models (VLM)
• Text-to-Image Models
• Audio-Visual Models
• Multimodal Transformers
• Cross-Modal Retrieval

---

LLM

• A reading list that from Ilya Sutskever
• ollama
• open-webui
• LLM Visualization
• LLM Transparency Tool

---

XAI (Explainable Artificial Intelligence)

• Model-agnostic (post-hoc explanations)
  • SHAP (SHapley Additive exPlanations)
  • Saliency Maps / Grad-CAM

---

Production Engineering & MLOps

Experiment Tracking & Monitoring

• Weights & Biases
• MLFlow
• TensorBoard
• Neptune
• Training-time Monitoring
• Visualizing Model Graphs & Gradients

Pre-deployment Quality & Validation

• Pre-release Quality Gates
• Packaging for Inference
• Model Deployment (ONNX, TorchScript)
• TensorRT
• Quantization & Pruning

Deployment & Operations

• Release & Rollout Strategies
• Production Monitoring
• A/B Testing
• Inference Optimization

GPU Optimization & Performance

• Maximize GPU Utilization
• Gradient Accumulation
• AMP Automatic Mixed Precision (amp)
• Gradient Checkpointing
• Memory Monitoring & Management
• Optimal Batch Size Selection
• Efficient Data Loading
• GPU Memory Optimization Techniques
• Clear Unused Variables & Cache

---

Infrastructure & Best Practices

• Data Versioning
• Logging and Debugging
• Project Structure & Best Practices
• Running Your PyTorch Projects on RunPod Using a Single Docker Image and GHCR
• GitHub Actions CI Setup, CUDA with CPU Fallback

---

Kaggle

• Kaggle Dataset Downloader & Management
  • Automated Kaggle dataset download with symlinks
  • Finding & searching datasets (CLI & web)
  • Running in Kaggle notebooks vs local/RunPod
  • Kaggle GPU specifications & optimization
  • Medical imaging, computer vision, 3D vision dataset examples

Datasets

• Waymo Open Dataset
• City Scapes Dataset
• KITTI — CC BY-NC-SA (share-alike, non-commercial)
• A2D2 (Audi) — CC BY-ND
• ADE20K — BSD-3, great for scene segmentation
• COCO / COCO-Stuff — CC BY
• Comma2k19 — MIT

---

Free Books and Online Courses

• Learning Deep Representations of Data Distributions
• The Principles of Diffusion Models
• 3D Scanning & Motion Capture (TUM-Matthias Niessner)
• Machine Learning for 3D Data
• Neural Radiance Fields | NeRF
• The Principles of Diffusion Models

---