Roadmap to become a Visual-SLAM developer in 2026
See the codeREADME
Visual-SLAM Developer Roadmap - 2026
Visual-SLAM is a special case of 'Simultaneous Localization and Mapping' in which a camera is used to gather exteroceptive sensory data.
Below is a set of topics you need to understand for Visual-SLAM, ranging from absolute-beginner level to getting ready to work as a Visual-SLAM engineer/researcher.
Visual-SLAM is often portrayed as a rather difficult topic - many think good C++ programming skills and a deep understanding of mathematics are necessary.
On the other hand, there are not many courses provided for beginners, especially in non-English languages.
I made this roadmap to share my thoughts and experience on studying Visual-SLAM, and hopefully beginners can get a grasp of where to start.
Purpose of these Roadmaps
The purpose of these roadmaps is to give you an idea about the general overview of Visual-SLAM, and to guide you if you are confused about where to start.
Note to Beginners
Acknowledge that SLAM has a relatively high entry barrier - it's not because it requires understanding difficult mathematics, but because it requires equipping yourself with various types of skills. Don't feel overwhelmed - you don't need to learn everything if you are just getting started. Instead, enjoy the journey itself and progress topic by topic. The result will be very rewarding.
Table of Contents
| Level | Topic | Focus |
|---|---|---|
| 1 | Beginner | Math, programming, and camera/image fundamentals |
| 2 | Getting Familiar | Geometry, optimization, and the anatomy of a SLAM system |
| 3 | Monocular SLAM | Classical monocular SLAM — feature-based, direct, semi-direct, SfM, dynamic scenes |
| 4 | RGB-D SLAM | Dense tracking and volumetric/surfel fusion with depth sensors |
| 5 | Deep Learning + SLAM | Learned frontends, differentiable backends, end-to-end systems, foundation-model & neural SLAM, scene understanding |
| 6 | VIO / VINS | Fusing cameras with IMUs — filtering vs optimization |
| 7 | Stereo SLAM | Metric scale and depth from stereo pairs |
| 8 | Collaborative SLAM | Multi-robot mapping, inter-robot loop closure, map merging |
| 9 | LiDAR & Visual-LiDAR | LiDAR odometry and tight camera–LiDAR–IMU fusion |
| 10 | Event Camera SLAM | Asynchronous vision for HDR and high-speed motion |
| 11 | World Models & Spatial AI | From SLAM maps to learned world representations |
Level 1: Beginner
Programming
- C++: Pointer, OOP
- Python
- Bash/Linux: Basic terminal usage
Mathematics
- Basic Probability & Statistics: Gaussian distribution, Bayes' theorem
- Basic Linear Algebra: Vectors & Matrices, Determinant, Dot & Cross product, Rank, Inverse matrix, Transpose matrix, SVD, Eigenvalues/Eigenvectors
- Logarithm & Exponential
- Basic Calculus: Differentiation, Taylor expansion
Projective Geometry
- Pinhole camera model → Image projection
- Camera calibration: Intrinsic/Extrinsic parameters, Lens distortion
- Rigid body motion: Euler/Quaternion/Rotation Matrix, Projective space & Vanishing point, Homogeneous transformation
- Epipolar geometry → Essential & Fundamental matrix
- Triangulation
- Camera models beyond pinhole: Fisheye (Kannala-Brandt), double-sphere, omnidirectional; rolling-shutter awareness
Camera Device
Image Data
- Color image, Resolution, Grayscale image
- Thresholding, Gaussian blur
- Corner detector: Harris corner
- Edge detector: Sobel & Canny Edge
- Stereo vision, RGB-D, Disparity, Depth
Level 2: Getting Familiar with SLAM
Programming (core)
- C++: OOP, Modern C++, Data structures & Algorithms, Compilers, CMake/Makefile/Ninja, Design patterns, OpenCV C++
- C
- Git/GitHub
- OpenCV (opencv-python)
- Python: Deep learning, Graph plots, System scripts
- Bash/Linux: ssh, CLI text editor/Vim/tmux
- Math libraries: Eigen, Ceres-solver/GTSAM/g2o
- C++/Python interop: PyBind11, nanobind
- ROS/ROS2
- Docker
Programming (optional — engineering skills for SLAM jobs)
- Concurrency: SIMD-SSE/AVX/Neon, OpenMP, CUDA
- Edge deployment: TensorRT/ONNX export of learned frontends, Jetson benchmarking
- Mobile: Android (Java/Kotlin), iOS (Objective-C/Swift)
- C#: Unity AR, Microsoft HoloLens
- CI/CD: GitHub Actions
- Simulation: Gazebo, Isaac Sim
Image Processing
- Keypoints → Detector/Descriptor
- Image pyramid, oFAST, rBRIEF
Local Feature Matching
Global Feature Matching
Feature Tracking
Multiple View Geometry
- 2D-2D correspondence: Essential/Fundamental, Homography
- 2D-3D correspondence: P3P, PnP, SVD
- 3D-3D correspondence: ICP
Outlier Rejection
- RANSAC, PROSAC, M-Estimator, MAXCON, Convex relaxation
- Robust pose-graph optimization: Switchable constraints, Dynamic covariance scaling (DCS), Pairwise consistency maximization (PCM)
Certifiably Optimal Algorithms
| System | Author/Year | Key Concepts |
|---|---|---|
| SE-Sync | Rosen 2019 | Certifiable pose graph optimization via SDP + Riemannian opt (arXiv 2016, IJRR 2019) |
| TEASER++ | Yang 2020 | Point cloud registration, 90%+ outlier robust, TLS + Max Clique (T-RO/RSS 2020) |
| GNC | Yang 2020 | Graduated Non-Convexity, continuation from convex → robust cost |
| QUASAR | Yang 2019 | Certifiably optimal rotation search (Wahba problem with outliers), quaternion QCQP + SDP relaxation |
Least Squares Optimization
- Reprojection error, Bundle adjustment
- Non-linear optimization, Lie algebra
- Lie groups: SO(3), SE(3)
- Gauss-Newton, Levenberg-Marquardt
- Pose graph optimization
- Schur complement / Sparsity
Motion Model
- Proprioceptive sensor: IMU, Wheel
- Odometry (pose)
Observation Model
- Exteroceptive sensor: Camera, LiDAR
- Landmark (Map)
- Joint optimization, MLE & MAP
Factor Graph Optimization
- Factor graph: SLAM as a bipartite graph of variables (poses, landmarks) and factors (measurements)
- MAP inference as sparse nonlinear least squares; variable elimination and the Bayes tree
- Incremental smoothing: iSAM / iSAM2
- Marginalization and fixed-lag smoothing
- Reference: Dellaert & Kaess, Factor Graphs for Robot Perception (2017); GTSAM tutorials
Gaussian Belief Propagation
| System | Author/Year | Key Concepts |
|---|---|---|
| FutureMapping 1 | Davison 2018 | Computational structure of Spatial AI, GBP for SLAM |
| FutureMapping 2 | Davison 2019 | GBP as core Spatial AI primitive, visual intro to GBP |
| BA on Graph Processor | Ortiz 2020 | Bundle Adjustment on Graphcore IPU, tile-based parallelism |
| DANCeRS | Patwardhan 2025 | GBP-based distributed consensus in robot swarms |
Mapping
Sensors
- Camera device: Wide/telecentric lens, Lens MTF, CCD/CMOS, Rolling/Global shutter, Exposure/ISO, Stereo vision, RGB-D, Structured light, Active IR/ToF
- LiDAR → Visual-LiDAR fusion
- IMU → VIO
- RADAR → Sensor fusion, Extended Kalman filter
- Sonar
- Multi-sensor calibration: Camera-IMU, Camera-LiDAR (Kalibr)
Evaluation
- Metrics: ATE (Absolute Trajectory Error), RPE (Relative Pose Error)
- Consistency: NEES (Normalized Estimation Error Squared)
- Datasets: KITTI, TUM RGB-D, EuRoC, TartanAir, TUM-VI, 4Seasons, Hilti SLAM Challenge, Newer College, Project Aria
- Tools: evo (trajectory evaluation)
Level 3: Monocular Visual-SLAM
Key Concepts
- VO vs SLAM — VO is local (no loop closure), SLAM includes global map + loop closure
- Scale ambiguity — Fundamental limitation of monocular SLAM; absolute scale is classically unrecoverable from geometry alone (learned metric-depth priors such as Metric3D or MASt3R can supply approximate scale)
- Covisibility graph — Shared map point visibility between keyframes; core data structure in ORB-SLAM
- Visual Place Recognition (VPR) — Recognizing previously visited places for loop closure
- Self-supervised depth — Learning monocular depth without ground truth (Monodepth2, Godard 2019)
Feature-based SLAM
| System | Author/Year | Key Concepts |
|---|---|---|
| Visual Odometry | Nistér 2004 | Five-point essential matrix solver, RANSAC, Triangulation, VO (local-only, no loop closure) |
| MonoSLAM | Davison 2007 | First real-time monocular SLAM, EKF-based, single camera, sparse 3D map, probabilistic feature initialization |
| PTAM | Klein & Murray 2007 | FAST feature, Tracking, Frontend/Backend separation, Parallel threads, Keyframe, Mapping, Bundle adjustment, Manual initialization |
| Visual-SLAM why filter? | Strasdat 2012 | Bundle adjustment, Scale-aware BA, Motion-only BA |
| ORB-SLAM | Mur-Artal 2015 | ORB keypoint, Automatic initialization (Homography vs Fundamental selection), Tracking thread, Local (covisibility-based) BA + global BA on loop closure, Local mapping, Large-scale, Loop closure, Bag of Visual Words, Global optimization, Covisibility graph, Map point management (culling, merging) |
| Pop-up SLAM | Yang 2016 | Line/Plane features |
| PL-SLAM | Pumarola 2017 | Point/Line features |
| ORB-SLAM2 | Mur-Artal 2017 | → Stereo SLAM, → RGB-D SLAM |
| CubeSLAM | Yang 2019 | Monocular 3D cuboid detection + SLAM, 9-DoF object representation |
| OpenVSLAM | Sumikura 2019 | ORB-based SLAM framework, perspective/fisheye/equirectangular camera models, map save/load + localization mode |
| Stella-VSLAM | Community 2021 | OpenVSLAM successor, license reboot (→ also in Level 7) |
| UcoSLAM | Muñoz-Salinas 2019 | Fiducial markers |
| DeepFusion | Laidlow 2019 | Dense monocular reconstruction, semi-dense MVS + CNN depth/gradient predictions, probabilistic fusion with learned uncertainties |
| ORB-SLAM3 | Campos 2020 | Monocular + Stereo + VIO, Multi-map, IMU integration |
| DXSLAM | Li 2020 | Deep features for SLAM |
| PyCuVSLAM | NVIDIA 2025 | Python + CUDA GPU-accelerated VSLAM toolkit (cuVSLAM wrapper; stereo/multi-camera VIO) |
Direct SLAM
| System | Author/Year | Key Concepts |
|---|---|---|
| DTAM | Newcombe 2011 | Dense mapping, Keyframe mapping, GPGPU |
| LSD-SLAM | Engel 2014 | Photometric error minimization, High gradient pixels/edges, Large scale, Loop closure, Pose graph optimization |
| DSO | Engel 2016 | Photometric bundle adjustment, Sliding window BA, No loop closure/global optimization |
| LDSO | Gao 2018 | DSO + Loop closure (BoW-based), addresses DSO's main weakness |
| CNN-SLAM | Tateno 2017 | Depth from LSD-SLAM + deep depth, Semantic label |
| DVSO | Yang 2018 | Deep single image depth estimation, StackNet |
| D3VO | Yang 2020 | Deep single image depth estimation, Deep pose, Deep aleatoric uncertainty |
Semi-direct (Hybrid)
| System | Author/Year | Key Concepts |
|---|---|---|
| SVO | Forster 2014 | FAST feature detection, sparse direct image alignment, depth filters |
| SVO2 | Forster 2017 | Multi-camera/Fisheye, Probabilistic depth estimation, Direct method convergence, Sparse method, Bundle adjustment |
| Stereo DSO | Wang 2017 | → Stereo SLAM |
| VI-DSO | von Stumberg 2018 | → VIO/VINS |
SfM Tools
| System | Author/Year | Key Concepts |
|---|---|---|
| COLMAP | Schönberger 2016 | De-facto standard incremental SfM + MVS pipeline (C++/CUDA, pycolmap bindings) |
| GLOMAP | Pan 2024 | Global SfM revisited — COLMAP-compatible, much faster mapping |
| InstantSfM | Zhong 2025 | GPU-native sparse-aware SfM pipeline, large speedups over COLMAP |
Dynamic Environment SLAM
| System | Author/Year | Key Concepts |
|---|---|---|
| DynaSLAM | Bescós 2018 | Mask R-CNN dynamic-object removal + background inpainting, ORB-SLAM2-based |
| DS-SLAM | Yu 2018 | Semantic segmentation (SegNet) + motion consistency check |
| MaskFusion | Rünz 2018 | RGB-D recognition, tracking and reconstruction of multiple moving objects |
| MID-Fusion | Xu 2019 | Octree-based object-level multi-instance dynamic RGB-D SLAM |
| VDO-SLAM | Zhang 2020 | Dynamic object-aware SLAM, joint camera + object motion estimation |
| DynaSLAM II | Bescós 2021 | Tightly-coupled multi-object tracking and SLAM |
| MonST3R | Zhang 2024 | DUSt3R-family pointmap estimation in the presence of motion |
Level 4: RGB-D Visual-SLAM
Key Concepts
- Depth from sensor — Structured light vs Active IR (ToF); metric scale for free, but range/material limitations
- Frame-to-model tracking — Aligning each frame against the accumulated model (ICP) instead of frame-to-frame
- TSDF vs Surfel maps — Volumetric signed-distance fusion (KinectFusion) vs point-based surfel fusion (ElasticFusion)
RGB-D Camera Devices
- Intel RealSense D series
- Orbbec Femto series (Azure Kinect successor), Orbbec Astra
- Luxonis OAK-D
- Legacy (discontinued): Microsoft Kinect v1/v2, Azure Kinect DK, Occipital Structure Core
GPGPU Programming
Systems
| System | Author/Year | Key Concepts |
|---|---|---|
| ICP | Besl & McKay 1992 | Iterative Closest Point, closest-point correspondence, closed-form rigid transform, local convergence (needs initialization), foundation of 3D-3D registration |
| DTAM | Newcombe 2011 | → see Level 3 Direct SLAM |
| KinectFusion | Newcombe 2011 | GPGPU, Tracking (project depth → 3D, surface normal, coarse-to-fine ICP), Mapping (volumetric integration, TSDF), Robust to small scene changes, Cannot model deformation, Map growth cubic, Room-size only |
| Double Window Optimisation | Strasdat 2011 | Inner window (local BA) + outer window (pose graph), covisibility graph, constant-time optimization |
| Kintinuous | Whelan 2012 | Volume shift, Geometric, Photometric, dBoW+SURF, Optimization, Loop closure |
| RGBD-SLAM-V2 | Endres 2013 | Tracking (color image, visual features, depth image, point cloud, transformation), Mapping (OctoMap 2013) |
| SLAM++ | Salas-Moreno 2013 | Object-oriented SLAM |
| DVO | Kerl 2013 | Keyframe, Depth, Direct method, Optimization, Loop closure |
| RTAB-Map | Labbé 2014 | Loop closure, Map merge, Multi-session memory management |
| MRS-Map | Stückler 2014 | Multi-resolution surfel maps in an octree, shape + color statistics per surfel, noise-aware RGB-D registration, real-time on CPU |
| ElasticFusion | Whelan 2015 | Active: frame-to-model tracking (photometric + geometric), joint optimization, fused surfel-based model reconstruction · Inactive: local loop closure (model-to-model local surface, submodel separation), global loop closure (randomised fern encoding, non-rigid space deformation) |
| DynamicFusion | Newcombe 2015 | 6D motion field, Deformable scene |
| ORB-SLAM2 (RGB-D mode) | Mur-Artal 2017 | Bundle adjustment, Sparse reconstruction (→ also in Level 3) |
| BundleFusion | Dai 2016 | Local-to-global optimization, Sparse RGB feature, Coarse global pose estimation, Fine pose refinement (geometric + photometric) |
| SemanticFusion | McCormac 2016 | Deep Learning CNN, Deep Semantic SLAM |
| InfiniTAM v3 | Prisacariu 2017 | Tracking (scene raycast, depth image, RGB image), Relocalization (random ferns), Mapping (TSDF reconstruction, voxel hashing, surfel reconstruction) |
| Fusion++ | McCormac & Clark 2018 | Deep Learning CNN, Mask-RCNN instance segmentation, Object-level SLAM, No prior, Object-level TSDF reconstruction |
| PointFusion / DenseFusion | Xu 2018 / Wang 2019 | RGB-D object 6-DoF pose estimation, point cloud + image feature fusion (object frontend for object-level SLAM) |
| BAD SLAM | Schöps 2019 | Direct RGB-D bundle adjustment, surfel map, real-time GPU BA, ETH3D benchmark |
| RTAB-Map (RGB-D / LiDAR) | Labbé 2019 | Multi-sensor RGB-D/LiDAR support, light-source detection (2016) |
| MoreFusion | Wada 2020 | DL instance segmentation, Object-level volumetric fusion, Volumetric pose prediction, 3D scene reconstruction, Collision-based refinement, Semantic SLAM, Object pose estimation, CAD object fitting |
| NodeSLAM | Sucar 2020 | Occupancy VAE, Object-level SLAM (→ also in Level 5 Latent Representation) |
| DSP-SLAM | Wang (UCL) 2021 | DeepSDF shape prior + ORB-SLAM2, object-level dense reconstruction (mono/stereo/LiDAR) |
Level 5: Applying Deep Learning
Key Concepts
- Learned vs hand-crafted — Replacing individual classical modules (features, depth, matching) with networks vs end-to-end learning
- Differentiability — Making classical optimization (RANSAC, BA) differentiable so it can be trained through
- Foundation models — Large pretrained models (CLIP, SAM, DUSt3R-family) as reusable perception backbones
Level 5 is organized into five pillars: A. Frontend — learned perception components replacing hand-crafted modules B. Backend — learned/certifiable optimization replacing classical solvers C. Systems — end-to-end deep VO/SLAM pipelines D. Scene Understanding — semantic, language, and relational reasoning on SLAM maps E. Foundation-Model & Neural SLAM — pointmap transformers, NeRF- and 3DGS-based dense SLAM systems
A. Deep Frontend — Perception
Feature Detection & Matching
| System | Author/Year | Key Concepts |
|---|---|---|
| NetVLAD | Arandjelović 2016 | VLAD, place recognition |
| SuperPoint | DeTone 2017 | Homographic Adaptation, Self-supervised, VGG encoder + detector/descriptor heads |
| HardNet | Mishchuk 2017 | Learned local descriptor |
| R2D2 | Revaud 2019 | Repeatable + Reliable detector/descriptor, explicit repeatability/reliability maps |
| KeyNet | Barroso-Laguna 2019 | Learned keypoint detector |
| HF-Net | Sarlin 2019 | Global feature, Local feature, Visual localization |
| SuperGlue | Sarlin 2020 | Self/Cross-attention GNN, Sinkhorn optimal assignment, dustbin for outliers |
| DISK | Tyszkiewicz 2020 | Policy gradient (RL) training, match success/failure as reward |
| Patch NetVLAD | Hausler 2021 | Multi-scale patch-level VLAD |
| LoFTR | Sun 2021 | Detector-free, Transformer coarse-to-fine dense matching |
| LightGlue | Lindenberger 2023 | Adaptive depth/width, 5-10× faster than SuperGlue |
| XFeat | Potje 2024 | 0.3M params, 1400 FPS (RTX 4090), 64-dim descriptor, embedded-friendly |
| RoMa | Edstedt 2024 | DINOv2 foundation feature + coarse-to-fine dense matching |
| DeDoDe | Edstedt 2024 | Joint detect-and-describe in one stage |
| RoMa v2 | Edstedt 2025 | Harder-better-faster-denser dense feature matching |
Depth Estimation
| System | Author/Year | Key Concepts |
|---|---|---|
| MonoDepth | Godard 2016 | Left-Right photometric consistency, self-supervised |
| MiDaS | Ranftl 2020 | Multi-dataset mixing, scale-and-shift invariant loss, relative depth |
| DPT | Ranftl 2021 | Dense Prediction Transformer (ViT backbone), global context |
| ZoeDepth | Bhat 2023 | Zero-shot metric depth, Metric Bins Module |
| Metric3D | Yin 2023 | Camera intrinsic-conditioned metric depth, Canonical Camera Space |
| Depth Anything | Yang 2024 | 62M images, foundation model for monocular depth |
| Depth Anything V2 | Yang 2024 | Improved with synthetic data, better edge preservation |
| Depth Anything 3 | Lin 2025 | Any-view geometry from arbitrary inputs, depth-ray prediction target, single plain transformer (DINOv2), teacher-student training |
| Marigold | Ke 2024 | Stable Diffusion for depth, fine detail, uncertainty via sampling |
| Align3R | Lu 2025 | Video temporal consistency, DUSt3R-based, CVPR 2025 Highlight |
| Masked Depth Modeling (LingBot-Depth) | Tan 2026 | Fixes RGB-D failures on glass/mirrors/metal |
Optical Flow & Scene Flow
| System | Author/Year | Key Concepts |
|---|---|---|
| FlowNet | Dosovitskiy 2015 | First end-to-end deep optical flow (SimpleNet / CorrNet) |
| FlowNet 2.0 | Ilg 2017 | Stacked networks, classical-level accuracy |
| PWC-Net | Sun 2018 | Pyramid-Warping-Cost volume, coarse-to-fine, 8.4M params |
| FlowNet3D | Liu 2019 | Point cloud scene flow, PointNet++ based |
| RAFT | Teed 2020 | All-Pairs Correlation + iterative ConvGRU update, ECCV Best Paper |
| RAFT-3D | Teed 2021 | Scene flow (3D motion) from RAFT |
| FlowFormer | Huang 2022 | Transformer on cost volume tokens, global context |
| SEA-RAFT | Wang 2024 | Efficient RAFT variant for real-time |
Camera Pose Regression & Relocalization
| System | Author/Year | Key Concepts |
|---|---|---|
| PoseNet | Kendall 2015 | CNN-based 6-DoF pose regression (APR), GoogLeNet backbone |
| DSAC | Brachmann 2017 | Differentiable RANSAC, Scene Coordinate Regression (SCR) |
| DSAC++ | Brachmann 2018 | Self-supervision, RGB-D support |
| CNN Pose Regression Limitations | Sattler 2019 | Pose regression ≈ image retrieval performance |
| LM-Reloc | von Stumberg 2020 | Deep direct relocalization |
| DSAC* | Brachmann 2021 | Visual relocalization from RGB/RGB-D, improved learning stability (TPAMI) |
| ACE | Brachmann 2023 | Accelerated Coordinate Encoding, 5-min training per scene |
| ACE Zero | Brachmann 2024 | Zero-shot SCR, no pre-built 3D map needed |
| ACE-G | Bruns 2025 | Generalizable SCR via query pretraining, new scenes without fine-tuning |
| ACE-SLAM | Alzugaray 2025 | Neural implicit real-time SLAM, network weights = map |
| hloc | Sarlin 2019 | Toolbox implementing HF-Net's hierarchical localization: coarse (NetVLAD) → fine (SuperGlue) |
Object Detection & Segmentation for SLAM
| System | Author/Year | Key Concepts |
|---|---|---|
| YOLO (v1→v11) | Redmon 2016→2024 | Real-time object detection, Ultralytics ecosystem |
| DETR | Carion 2020 | Transformer detection, anchor-free, no NMS |
| RT-DETR | Zhao (Baidu) 2023 | Real-time DETR, YOLO-speed + Transformer quality |
| RF-DETR | Robinson 2025 | Weight-sharing NAS over DETRs, accuracy-latency Pareto tuning, first real-time detector past 60 AP on COCO |
| SAM | Kirillov 2023 | Segment Anything, prompt-based, Foundation Model |
| SAM 2 | Meta 2024 | Video segmentation, Memory Attention, temporal consistency |
| SAM 3 | Carion 2025 | Promptable concept segmentation (noun-phrase / exemplar prompts), presence head, detector + memory-based video tracker |
| Grounding DINO | Liu 2023 | Text-prompted detection → SAM pipeline (Grounded SAM) |
| Open-YOLO 3D | Boudjoghra 2024 | 2D open-vocab detection → 3D instance seg, 16× faster |
B. Deep Backend — Optimization
Differentiable Bundle Adjustment
| System | Author/Year | Key Concepts |
|---|---|---|
| BA-Net | Tang 2019 | FPN + differentiable LM layer, end-to-end SfM (ICLR) |
| DROID-SLAM | Teed 2021 | Dense optical flow + differentiable dense BA, all-pixels reprojection |
| DPVO | Teed 2023 | Patch-based DROID-SLAM, 30+ FPS real-time |
| Theseus | Pineda (Meta) 2022 | Differentiable nonlinear optimization library (PyTorch) |
| Lietorch | Teed 2021 | Lie group operations for PyTorch (SE(3)/SO(3)) |
C. End-to-End Deep VO / SLAM Systems
Self-supervised & Learned VO
| System | Author/Year | Key Concepts |
|---|---|---|
| DeepVO | Wang 2017 | Supervised learning |
| SfM-Learner | Zhou 2017 | Unsupervised, deep depth + deep pose |
| DeMoN | Ummenhofer 2017 | Depth + Motion from two frames, encoder-decoder |
| UndeepVO | Li 2018 | Stereo self-supervised, absolute scale recovery |
| DeepTAM | Zhou 2018 | Deep tracking and mapping, cost volume based |
| DeepV2D | Teed 2018 | Iterative depth from video, differentiable geometry layers |
| Depth from Videos in the Wild | Gordon 2019 | Unconstrained video depth, learned camera intrinsics |
| Neural Ray Surfaces | Vasiljevic 2020 | Learned ray surface model, non-pinhole cameras |
| GradSLAM | Murthy 2020 | Differentiable SLAM framework (PyTorch, supports multiple SLAM backends) |
| DeepSLAM | Li 2020 | TrackingNet, MappingNet, LoopNet |
| MonoRec | Wimbauer 2021 | Self-supervised monocular 3D reconstruction, moving objects |
| TANDEM | Koestler 2021 | Real-time tracking + dense mapping via MVS depth, DSO-based |
Learning-based SLAM Systems
| System | Author/Year | Key Concepts |
|---|---|---|
| DROID-SLAM | Teed 2021 | Differentiable BA, dense optical flow, end-to-end learned |
| TartanVO | Wang 2021 | Generalizable visual odometry |
| DPV-SLAM | Lipson 2024 | DPVO + loop closure, full SLAM (ECCV 2024) |
| MAC-VO | Qiu 2024 | Learning-based VO, metric-aware |
| VoT | Yugay 2025 | Visual Odometry with Transformers (later retitled FVO) |
Latent Representation SLAM
| System | Author/Year | Key Concepts |
|---|---|---|
| CodeSLAM | Bloesch 2018 | Depth as 128-dim latent code, photometric BA on codes + poses |
| SceneCode | Zhi 2019 | Depth + semantic in single latent code, cross-modal constraints |
| DeepFactors | Czarnowski 2020 | Probabilistic depth codes + factor graph, GPU 30+ FPS |
| NodeSLAM | Sucar 2020 | Object-level DeepSDF codes, occupancy VAE per object |
| CodeMapping | Matsuki 2021 | Sparse SLAM + learned dense mapping, hybrid approach |
Neural Rendering (reference)
NeRF/3DGS-based SLAM systems → see Pillar E below
| System | Author/Year | Key Concepts |
|---|---|---|
| NeRF | Mildenhall 2020 | Neural Radiance Fields, novel view synthesis (foundational) |
| DIFIX3D+ | Wu 2025 | Single-step diffusion for 3D reconstruction artifact removal (post-processing) |
D. Scene Understanding
Benchmarks & Foundations
| System | Author/Year | Key Concepts |
|---|---|---|
| EFM3D | Straub (Meta) 2024 | Egocentric Foundation Model 3D benchmark, depth/surface/semantic from ego-video |
3D Scene Graph
| System | Author/Year | Key Concepts |
|---|---|---|
| Kimera / 3D Dynamic Scene Graph | Rosinol 2020 | Kimera-VIO, Kimera-Mesher, Kimera-PGMO, Kimera-Semantics, Kimera-DSG (stereo/mono visual-inertial pipeline) |
| Hydra | Hughes (MIT SPARK) 2022 | Real-time hierarchical Scene Graph (mesh→objects→places→rooms→buildings) |
| Hydra-Multi | Chang 2023 | Distributed multi-robot 3D Scene Graph |
| Clio | Maggio (MIT SPARK) 2024 | Open-set task-driven Scene Graph, CLIP embeddings per node |
| Khronos | Schmid (MIT SPARK) 2024 | Spatio-temporal Scene Graph, dynamic object history tracking |
| ConceptGraphs | Gu 2023 | Open-vocabulary 3D Scene Graph, SAM + CLIP + LLM relations |
Semantic / Language-Grounded SLAM
| System | Author/Year | Key Concepts |
|---|---|---|
| ConceptFusion | Jatavallabhula (MIT) 2023 | CLIP features fused into 3D map, open-vocabulary language queries |
| LERF | Kerr 2023 | Language Embedded Radiance Fields, DINO multi-scale, NeRF + CLIP |
| OpenScene | Peng (ETH) 2023 | Language features back-projected to 3D point clouds |
| SpatialLM | Mao 2025 | Point cloud → LLM, structured indoor modeling as Python scripts |
Also see: LEGS, OpenGS-SLAM (Pillar E above); Open-YOLO 3D (Level 5 Object Detection)
E. Foundation-Model & Neural-Representation SLAM
Foundation-Model SLAM
| System | Author/Year | Key Concepts |
|---|---|---|
| DUSt3R | Wang 2024 | Pointmap regression from image pairs, no calibration needed |
| MASt3R | Leroy 2024 | DUSt3R + local feature matching |
| MASt3R-SLAM | Murai 2024 | Real-time dense SLAM from MASt3R priors |
| VGGT | Wang (Meta) 2025 | Feed-forward inference of poses, depths, pointmaps, tracks from N views (CVPR 2025 Best Paper) |
| VGGT-SLAM | Maggio 2025 | Dense RGB SLAM optimized on the SL(4) manifold, VGGT frontend |
| VGGT-SLAM 2.0 | Maggio 2026 | Real-time dense feed-forward scene reconstruction |
| VGGT-Geo | Qin 2026 | Probabilistic geometric fusion of VGGT priors for dense indoor SLAM |
| IGGT | Li 2025 | Instance-grounded geometry transformer — unified 3D reconstruction + instance-level understanding |
| AMB3R | Wang 2025 | Accurate feed-forward metric-scale 3D reconstruction with backend, SfM/SLAM support |
| MASt3R-Fusion | Zhou 2025 | MASt3R feed-forward visual model + IMU + GNSS fusion |
NeRF-based
| System | Author/Year | Key Concepts |
|---|---|---|
| iMAP | Sucar 2021 | First NeRF-SLAM, single MLP, real-time tracking/mapping |
| BARF | Lin 2021 | Bundle-Adjusting NeRF, coarse-to-fine positional encoding, joint pose+NeRF opt (not full SLAM — pose+NeRF co-optimization) |
| NICE-SLAM | Zhu & Peng 2022 | Hierarchical feature grid (coarse/mid/fine), scalable |
| Co-SLAM | Wang 2023 | Hash grid (Instant-NGP) + coordinate encoding, 5-10× faster than NICE-SLAM |
| ESLAM | Johari 2023 | Tri-plane representation, O(N²) vs O(N³) memory |
| Point-SLAM | Sandström 2023 | Neural point cloud based |
| NeRF-SLAM | Rosinol 2023 | NeRF + classical SLAM pipeline |
| NICER-SLAM | Zhu 2024 | RGB-only NeRF-SLAM (no depth sensor), monocular depth integration |
| vMAP | Kong 2023 | Object-level NeRF-SLAM, per-object neural fields |
| GO-SLAM | Zhang 2023 | Global optimization + NeRF-SLAM, loop closure + global BA |
3DGS-based
| System | Author/Year | Key Concepts |
|---|---|---|
| SplaTAM | Keetha 2024 | Among the first 3DGS SLAM systems (concurrent with GS-SLAM, MonoGS), RGB-D, silhouette-guided densification |
| MonoGS | Matsuki 2024 | First monocular 3DGS SLAM (CVPR 2024 highlight), direct rasterization-based tracking, analytic camera Jacobians |
| GS-ICP SLAM | Ha 2024 | Gaussian-to-Gaussian ICP (Mahalanobis distance), geometric tracking |
| Photo-SLAM | Huang 2024 | Explicit geometry + implicit appearance (MLP color), anti-aliasing |
| RTG-SLAM | Peng 2024 | Real-time focus, adaptive Gaussian budget, Jetson Orin 25 FPS |
| EGG-Fusion | Pan 2025 | Geometry-aware Gaussian surfel fusion on the fly, information-filter-based, real-time |
| Online 3DGS Modeling | Lee 2025 | Online 3D Gaussian Splatting modeling with novel view selection |
| ActiveSplat | Li 2025 | Active mapping with 3DGS + Voronoi-based path planning |
| OpenGS-SLAM | Yang 2025 | Open-set dense semantic 3DGS SLAM, object-level scene understanding |
| LEGS | Yu 2024 | Language Embedded Gaussian Splats, real-time language-queryable 3D |
Level 6: VIO / VINS
Key Concepts
- Tightly-coupled vs Loosely-coupled — Joint vs separate optimization of visual and inertial measurements
- Filter-based vs Optimization-based — EKF approaches vs nonlinear optimization (BA)
- IMU preintegration — Integrating IMU measurements between keyframes (Lupton 2012; on-manifold formulation: Forster 2015)
- IMU noise model — Bias, random walk, Allan variance
- Observability — 4 unobservable DoF in VIO (3-DoF global translation + yaw); scale becomes additionally unobservable under constant-acceleration motion
- Deployed VIO — Commercial XR stacks (Meta Quest, ARKit/ARCore) are the highest-volume deployed VIO systems — worth studying as case studies
Foundations
| Resource | Author/Year | Key Concepts |
|---|---|---|
| Introduction to Inertial Navigation | Woodman 2007 | IMU fundamentals, coordinate frames, error sources — essential prerequisite |
| IMU Preintegration on Manifold | Forster 2015 | On-manifold preintegration, bias correction without re-integration |
| Quaternion kinematics for error-state KF | Solà 2017 | Quaternion math, error-state formulation |
Filter-based
| System | Author/Year | Key Concepts |
|---|---|---|
| MSCKF | Mourikis 2007 | Multi-State Constraint KF, efficient VIO without landmarks in state |
| ROVIO | Bloesch 2015 | Robocentric VIO, direct photometric tracking + EKF |
| OpenVINS | Geneva 2020 | Open-source MSCKF, modular, extensible |
Optimization-based
| System | Author/Year | Key Concepts |
|---|---|---|
| OKVIS | Leutenegger 2015 | Keyframe-based, tightly-coupled, sliding window optimization |
| VINS-Mono | Qin 2018 | Tightly-coupled, relocalization, loop closure, pose graph optimization |
| VI-DSO | von Stumberg 2018 | Direct sparse VIO, dynamic marginalization, photometric error |
| VINS-Fusion | Qin 2019 | Stereo + GPS fusion extension |
| maplab | Schneider 2018 | Multi-session visual-inertial mapping framework |
| Kimera-VIO | Rosinol 2020 | Fast VIO frontend for Kimera pipeline, structureless vision factors |
| Basalt | Usenko 2020 | Non-linear factor recovery (NFR) of marginalization priors, visual-inertial odometry + mapping |
| ORB-SLAM3 | Campos 2020 | VIO mode, multi-map, IMU initialization |
| DM-VIO | von Stumberg 2022 | Direct (DSO-based) monocular VIO, delayed marginalization, pose-graph BA for IMU initialization |
| OKVIS2 | Leutenegger 2022 | Multi-session, improved marginalization |
| AirVO | Xu 2023 | Point-line VIO, illumination-robust |
| OKVIS2-X | Boche & Leutenegger 2025 | Multi-sensor SLAM (Visual+Inertial+Depth+LiDAR+GNSS), dense volumetric occupancy maps, submapping for large-scale (9km+), EuRoC/Hilti22 SOTA |
Level 7: Stereo SLAM
Key Concepts
- Stereo rectification — Epipolar alignment for efficient disparity search
- Disparity vs Depth — d = f·B/Z, baseline determines depth range/accuracy
- Scale observability — Stereo provides metric scale (unlike monocular)
Systems
| System | Author/Year | Key Concepts |
|---|---|---|
| S-PTAM | Pire 2017 | Stereo PTAM, ROS-compatible, real-time |
| ORB-SLAM2 (stereo) | Mur-Artal 2017 | Stereo + RGB-D modes, loop closure, relocalization |
| StereoMSCKF | Sun 2018 | MSCKF with stereo, efficient for resource-constrained platforms |
| RTAB-Map | Labbé 2019 | Multi-sensor (stereo/RGB-D/LiDAR), memory management, large-scale (→ also in Level 4) |
| ORB-SLAM3 (stereo) | Campos 2020 | Multi-map, Atlas, stereo + IMU |
| Stella-VSLAM | Community 2021 | OpenVSLAM successor, stereo support (→ also in Level 3) |
| Stereo DSO | Wang 2017 | Direct sparse stereo odometry, large-scale (DSO extension) |
Level 8: Collaborative / Multi-Robot SLAM
Key Concepts
- Centralized vs Decentralized — Single server vs peer-to-peer map merging
- Inter-robot loop closure — Place recognition across robots with different viewpoints
- Communication constraints — Bandwidth-limited map sharing, sparse descriptors
- Map merging — Aligning submaps from different robots into a global map
Systems
| System | Author/Year | Key Concepts |
|---|---|---|
| C2TAM | Riazuelo 2014 | Cloud-based collaborative monocular SLAM |
| CCM-SLAM | Schmuck & Chli 2019 | Centralized collaborative monocular SLAM, robust to comm failures |
| DOOR-SLAM | Lajoie 2020 | Distributed, outlier-resilient SLAM with pairwise consistency |
| Kimera-Multi | Tian 2022 | Distributed multi-robot metric-semantic SLAM, mesh reconstruction |
| Swarm-SLAM | Lajoie 2024 | Decentralized, sparse, scalable C-SLAM, supports LiDAR/stereo/RGB-D |
| CoPeD | Zhou 2024 | Multi-robot collaborative perception dataset (real-world, aerial + ground robots) |
| maplab 2.0 | Cramariuc 2023 | Multi-session, multi-robot visual-inertial mapping |
Level 9: LiDAR & Visual-LiDAR Fusion SLAM
Key Concepts
- LiDAR-Visual-Inertial (LVI) — Triple fusion for robust outdoor SLAM
- Tightly-coupled LiDAR-camera — Joint optimization of point cloud and visual features
- Direct LiDAR-camera alignment — Photometric/geometric alignment without feature extraction
- Degradation handling — Graceful fallback when one modality fails (e.g., LiDAR in rain, camera in darkness)
- Range image — 2D projection of LiDAR scans for efficient processing (SuMa, RangeNet++)
LiDAR / LiDAR-Inertial SLAM
| System | Author/Year | Key Concepts |
|---|---|---|
| LOAM | Zhang 2014 | LiDAR odometry and mapping (foundational), edge + planar features |
| SuMa | Behley (Bonn) 2018 | Surfel-based LiDAR SLAM, projective ICP on range images |
| SuMa++ | Chen (Bonn) 2019 | SuMa + RangeNet++ semantics, semantic ICP weighting, dynamic object filtering |
| LIO-SAM | Shan 2020 | Tightly-coupled LiDAR-inertial, factor graph, GPS fusion |
| FAST-LIO2 | Xu 2022 | Direct LiDAR-inertial, ikd-Tree, extremely fast |
| PIN-SLAM | Pan (Bonn) 2024 | Neural point cloud LiDAR SLAM, point-to-SDF registration, elastic map deformation for loop closure |
Visual-LiDAR Fusion SLAM
| System | Author/Year | Key Concepts |
|---|---|---|
| LVI-SAM | Shan 2021 | LiDAR-Visual-Inertial via factor graph, LIO-SAM + VINS-Mono |
| R3LIVE | Lin 2022 | Real-time LiDAR-Visual-Inertial, dense RGB point cloud map |
| R3LIVE++ | Lin 2023 | Improved R3LIVE with mesh reconstruction |
| FAST-LIVO | Zheng 2022 | FAST-LIO + direct visual odometry, tightly-coupled LVI |
| FAST-LIVO2 | Zheng 2024 | Improved, sequential image processing, direct photometric fusion |
| OKVIS2-X | Boche 2025 | Visual+Inertial+Depth+LiDAR+GNSS configurable (also in Level 6) |
Resources
| Resource | Author/Year | Key Concepts |
|---|---|---|
| Multi-Sensor Fusion SLAM Survey | Zhu 2024 | Camera + LiDAR + IMU fusion SLAM — comprehensive survey |
Level 10: Event Camera SLAM
Key Concepts
- Event cameras (DVS) — Asynchronous per-pixel brightness change detection, μs temporal resolution
- Advantages — HDR (140dB+), no motion blur, low latency, low power
- Challenges — No absolute intensity, sparse asynchronous output, requires new algorithms
- Event representations — Event frames, time surfaces, voxel grids, spike tensors
Foundations
| Resource | Author/Year | Key Concepts |
|---|---|---|
| Event-based Vision Survey | Gallego 2020 | Comprehensive survey of event camera algorithms |
| Awesome-Event-based-SLAM | KwanWaiPang | Curated GitHub list of event-based SLAM papers |
Systems
| System | Author/Year | Key Concepts |
|---|---|---|
| EVO | Rebecq 2017 | Event-based Visual Odometry, 3D reconstruction from events |
| ESVO | Zhou 2021 | Event-based Stereo Visual Odometry |
| Ultimate-SLAM | Vidal 2018 | Events + frames + IMU fusion |
| EKLT | Gehrig 2020 | Event-based KLT feature tracking |
| ESVIO | Chen 2023 | Event-based Stereo VIO |
| EDS | Hidalgo-Carrió 2022 | Event-aided direct sparse odometry |
| DEVO | Klenk 2024 | Deep event-based visual odometry, DPVO-style patch-based, trained on simulated events |
| VIO-GO | Sakhrieh 2025 | Event-based VIO with optimized parameters for HDR scenarios |
Level 11: World Models & Spatial AI
Key Concepts
- World model — A learned generative model of environment dynamics, usable for prediction and planning
- VLM vs VLA — Vision-language models reason about images; vision-language-action models additionally output robot actions
- Spatial AI — The convergence of SLAM, scene understanding, and learned world representations (Davison's FutureMapping vision)
World Models
| System | Author/Year | Key Concepts |
|---|---|---|
| GAIA-1 | Wayve 2023 | Driving World Model, action-conditioned future scene generation |
| Sora / DiT | OpenAI 2024 | Diffusion Transformer, spacetime patches, emergent 3D understanding |
| NVIDIA Cosmos | NVIDIA 2025 | World Foundation Model platform for Physical AI, synthetic data for AV/robots |
| World Labs / Marble | Fei-Fei Li 2025 | Generative 3D worlds (persistent Gaussian-splat scenes) from image/video/text prompts |
| WorldVLA | Cen (Alibaba) 2025 | Autoregressive action world model, learns physics for action generation |
| SceneDINO | Jevtić 2025 | Feed-forward unsupervised semantic scene completion |
Generative 3D
| System | Author/Year | Key Concepts |
|---|---|---|
| DreamFusion | Poole 2023 | Text-to-3D via Score Distillation Sampling (SDS) + NeRF |
Vision-Language Models (VLM)
| System | Author/Year | Key Concepts |
|---|---|---|
| CLIP | Radford (OpenAI) 2021 | Contrastive image-text pretraining, 400M pairs, zero-shot |
| SigLIP | Zhai (Google) 2023 | Sigmoid loss CLIP, more efficient, better at small model sizes |
| BLIP-2 | Li (Salesforce) 2023 | Q-Former bridges frozen LLM + image encoder |
| LLaVA | Liu 2023 | LLaMA + vision, conversational VLM |
Vision-Language-Action Models (VLA)
| System | Author/Year | Key Concepts |
|---|---|---|
| RT-2 | Brohan (DeepMind) 2023 | Robot actions as text tokens, emergent generalization |
| OpenVLA | Kim 2024 | Open-source VLA, SigLIP + Llama 7B + Action Head |
| NaVILA | Cheng 2024 | Legged/wheeled-robot vision-language-action model for navigation |
Resources
| Resource | Author/Year | Key Concepts |
|---|---|---|
| Awesome-Transformer-based-SLAM | KwanWaiPang | Curated GitHub list of Transformer-based SLAM methods |
Study Resources
YouTube Lecture Series
| Lecture | Instructor | Link |
|---|---|---|
| SLAM & Photogrammetry | Cyrill Stachniss (Uni Bonn) | YouTube Playlist |
| First Principles of Computer Vision | Shree Nayar (Columbia) | YouTube Channel |
| Multiple View Geometry | Daniel Cremers (TU Munich) | YouTube Playlist |
Books
| Book | Author | Key Topics |
|---|---|---|
| Introduction to Visual SLAM | Xiang Gao et al. | VO, optimization, Lie algebra, backend, loop closure — best entry-level SLAM book |
| Photogrammetric Computer Vision | Wolfgang Förstner & Bernhard Wrobel | Camera geometry, estimation, 3D reconstruction — mathematically rigorous |
| Multiple View Geometry in Computer Vision | Richard Hartley & Andrew Zisserman | Epipolar geometry, trifocal tensor, reconstruction — THE bible |
| Computer Vision: Algorithms and Applications | Richard Szeliski | Feature detection, stereo, motion, 3D — comprehensive reference (2nd ed. free PDF) |
| State Estimation for Robotics | Timothy Barfoot | Estimation theory, Lie groups, batch/recursive estimation — free PDF (2nd ed.) |
| Probabilistic Robotics | Thrun, Burgard & Fox | Bayes filters, EKF/particle-filter SLAM — the classical probabilistic foundation |
| Factor Graphs for Robot Perception | Frank Dellaert & Michael Kaess | Factor graphs, elimination, iSAM2 — the backend bible (free PDF) |
| SLAM Handbook | Carlone, Kim, Barfoot, Cremers, Dellaert (eds.) | From localization and mapping to spatial intelligence — free community book (2024-25) |
Surveys
| Survey | Author/Year | Key Concepts |
|---|---|---|
| Past, Present, and Future of SLAM | Cadena 2016 | The canonical orientation survey — robust perception age, open problems |
| Event-based Vision Survey | Gallego 2020 | Event cameras and algorithms (→ also in Level 10) |
Code & Practice
| Resource | Link |
|---|---|
| SLAM Zero-to-Hero code exercises | GitHub — Docker-based hands-on exercises for this roadmap's topics (feature detection, epipolar geometry, RANSAC, ICP, g2o/GTSAM/Ceres) and systems (ORB-SLAM2, Basalt, Kimera, FAST-LIO2, MASt3R-SLAM, ...); individual exercises are linked from the matching study notes |
| changh95/slam_lecture_codes | GitHub — Full SLAM lecture code collection |
Wrap Up
If you think any part of the roadmap can be improved, please open a PR or submit an issue. I will continue to improve this, so you may want to watch/star this repository and revisit it later.
Contribution
- Open pull request with improvements
- Discuss ideas in issues
- Spread the word
- Reach out to me directly at hyunggi.chang95[at]gmail.com
Discussion
To discuss any topics or ask questions, please use the issue tab.
License
This project is licensed under the MIT License:
Copyright © 2026 Hyunggi Chang.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
awesome
awesome-list
computer-vision
deep-learning
rgb-d
roadmap
robotics
slam
vio
visual-inertial-odometry
visual-slam
Contributors
changh95
59 commits
Similar repos
Languages
Astro
69.3%
JavaScript
22.8%
CSS
6.0%
TypeScript
1.9%
Roadmap to become a Visual-SLAM developer in 2026
See the codeREADME
Visual-SLAM Developer Roadmap - 2026
Visual-SLAM is a special case of 'Simultaneous Localization and Mapping' in which a camera is used to gather exteroceptive sensory data.
Below is a set of topics you need to understand for Visual-SLAM, ranging from absolute-beginner level to getting ready to work as a Visual-SLAM engineer/researcher.
Visual-SLAM is often portrayed as a rather difficult topic - many think good C++ programming skills and a deep understanding of mathematics are necessary.
On the other hand, there are not many courses provided for beginners, especially in non-English languages.
I made this roadmap to share my thoughts and experience on studying Visual-SLAM, and hopefully beginners can get a grasp of where to start.
Purpose of these Roadmaps
The purpose of these roadmaps is to give you an idea about the general overview of Visual-SLAM, and to guide you if you are confused about where to start.
Note to Beginners
Acknowledge that SLAM has a relatively high entry barrier - it's not because it requires understanding difficult mathematics, but because it requires equipping yourself with various types of skills. Don't feel overwhelmed - you don't need to learn everything if you are just getting started. Instead, enjoy the journey itself and progress topic by topic. The result will be very rewarding.
Table of Contents
| Level | Topic | Focus |
|---|---|---|
| 1 | Beginner | Math, programming, and camera/image fundamentals |
| 2 | Getting Familiar | Geometry, optimization, and the anatomy of a SLAM system |
| 3 | Monocular SLAM | Classical monocular SLAM — feature-based, direct, semi-direct, SfM, dynamic scenes |
| 4 | RGB-D SLAM | Dense tracking and volumetric/surfel fusion with depth sensors |
| 5 | Deep Learning + SLAM | Learned frontends, differentiable backends, end-to-end systems, foundation-model & neural SLAM, scene understanding |
| 6 | VIO / VINS | Fusing cameras with IMUs — filtering vs optimization |
| 7 | Stereo SLAM | Metric scale and depth from stereo pairs |
| 8 | Collaborative SLAM | Multi-robot mapping, inter-robot loop closure, map merging |
| 9 | LiDAR & Visual-LiDAR | LiDAR odometry and tight camera–LiDAR–IMU fusion |
| 10 | Event Camera SLAM | Asynchronous vision for HDR and high-speed motion |
| 11 | World Models & Spatial AI | From SLAM maps to learned world representations |
Level 1: Beginner
Programming
- C++: Pointer, OOP
- Python
- Bash/Linux: Basic terminal usage
Mathematics
- Basic Probability & Statistics: Gaussian distribution, Bayes' theorem
- Basic Linear Algebra: Vectors & Matrices, Determinant, Dot & Cross product, Rank, Inverse matrix, Transpose matrix, SVD, Eigenvalues/Eigenvectors
- Logarithm & Exponential
- Basic Calculus: Differentiation, Taylor expansion
Projective Geometry
- Pinhole camera model → Image projection
- Camera calibration: Intrinsic/Extrinsic parameters, Lens distortion
- Rigid body motion: Euler/Quaternion/Rotation Matrix, Projective space & Vanishing point, Homogeneous transformation
- Epipolar geometry → Essential & Fundamental matrix
- Triangulation
- Camera models beyond pinhole: Fisheye (Kannala-Brandt), double-sphere, omnidirectional; rolling-shutter awareness
Camera Device
Image Data
- Color image, Resolution, Grayscale image
- Thresholding, Gaussian blur
- Corner detector: Harris corner
- Edge detector: Sobel & Canny Edge
- Stereo vision, RGB-D, Disparity, Depth
Level 2: Getting Familiar with SLAM
Programming (core)
- C++: OOP, Modern C++, Data structures & Algorithms, Compilers, CMake/Makefile/Ninja, Design patterns, OpenCV C++
- C
- Git/GitHub
- OpenCV (opencv-python)
- Python: Deep learning, Graph plots, System scripts
- Bash/Linux: ssh, CLI text editor/Vim/tmux
- Math libraries: Eigen, Ceres-solver/GTSAM/g2o
- C++/Python interop: PyBind11, nanobind
- ROS/ROS2
- Docker
Programming (optional — engineering skills for SLAM jobs)
- Concurrency: SIMD-SSE/AVX/Neon, OpenMP, CUDA
- Edge deployment: TensorRT/ONNX export of learned frontends, Jetson benchmarking
- Mobile: Android (Java/Kotlin), iOS (Objective-C/Swift)
- C#: Unity AR, Microsoft HoloLens
- CI/CD: GitHub Actions
- Simulation: Gazebo, Isaac Sim
Image Processing
- Keypoints → Detector/Descriptor
- Image pyramid, oFAST, rBRIEF
Local Feature Matching
Global Feature Matching
Feature Tracking
Multiple View Geometry
- 2D-2D correspondence: Essential/Fundamental, Homography
- 2D-3D correspondence: P3P, PnP, SVD
- 3D-3D correspondence: ICP
Outlier Rejection
- RANSAC, PROSAC, M-Estimator, MAXCON, Convex relaxation
- Robust pose-graph optimization: Switchable constraints, Dynamic covariance scaling (DCS), Pairwise consistency maximization (PCM)
Certifiably Optimal Algorithms
| System | Author/Year | Key Concepts |
|---|---|---|
| SE-Sync | Rosen 2019 | Certifiable pose graph optimization via SDP + Riemannian opt (arXiv 2016, IJRR 2019) |
| TEASER++ | Yang 2020 | Point cloud registration, 90%+ outlier robust, TLS + Max Clique (T-RO/RSS 2020) |
| GNC | Yang 2020 | Graduated Non-Convexity, continuation from convex → robust cost |
| QUASAR | Yang 2019 | Certifiably optimal rotation search (Wahba problem with outliers), quaternion QCQP + SDP relaxation |
Least Squares Optimization
- Reprojection error, Bundle adjustment
- Non-linear optimization, Lie algebra
- Lie groups: SO(3), SE(3)
- Gauss-Newton, Levenberg-Marquardt
- Pose graph optimization
- Schur complement / Sparsity
Motion Model
- Proprioceptive sensor: IMU, Wheel
- Odometry (pose)
Observation Model
- Exteroceptive sensor: Camera, LiDAR
- Landmark (Map)
- Joint optimization, MLE & MAP
Factor Graph Optimization
- Factor graph: SLAM as a bipartite graph of variables (poses, landmarks) and factors (measurements)
- MAP inference as sparse nonlinear least squares; variable elimination and the Bayes tree
- Incremental smoothing: iSAM / iSAM2
- Marginalization and fixed-lag smoothing
- Reference: Dellaert & Kaess, Factor Graphs for Robot Perception (2017); GTSAM tutorials
Gaussian Belief Propagation
| System | Author/Year | Key Concepts |
|---|---|---|
| FutureMapping 1 | Davison 2018 | Computational structure of Spatial AI, GBP for SLAM |
| FutureMapping 2 | Davison 2019 | GBP as core Spatial AI primitive, visual intro to GBP |
| BA on Graph Processor | Ortiz 2020 | Bundle Adjustment on Graphcore IPU, tile-based parallelism |
| DANCeRS | Patwardhan 2025 | GBP-based distributed consensus in robot swarms |
Mapping
Sensors
- Camera device: Wide/telecentric lens, Lens MTF, CCD/CMOS, Rolling/Global shutter, Exposure/ISO, Stereo vision, RGB-D, Structured light, Active IR/ToF
- LiDAR → Visual-LiDAR fusion
- IMU → VIO
- RADAR → Sensor fusion, Extended Kalman filter
- Sonar
- Multi-sensor calibration: Camera-IMU, Camera-LiDAR (Kalibr)
Evaluation
- Metrics: ATE (Absolute Trajectory Error), RPE (Relative Pose Error)
- Consistency: NEES (Normalized Estimation Error Squared)
- Datasets: KITTI, TUM RGB-D, EuRoC, TartanAir, TUM-VI, 4Seasons, Hilti SLAM Challenge, Newer College, Project Aria
- Tools: evo (trajectory evaluation)
Level 3: Monocular Visual-SLAM
Key Concepts
- VO vs SLAM — VO is local (no loop closure), SLAM includes global map + loop closure
- Scale ambiguity — Fundamental limitation of monocular SLAM; absolute scale is classically unrecoverable from geometry alone (learned metric-depth priors such as Metric3D or MASt3R can supply approximate scale)
- Covisibility graph — Shared map point visibility between keyframes; core data structure in ORB-SLAM
- Visual Place Recognition (VPR) — Recognizing previously visited places for loop closure
- Self-supervised depth — Learning monocular depth without ground truth (Monodepth2, Godard 2019)
Feature-based SLAM
| System | Author/Year | Key Concepts |
|---|---|---|
| Visual Odometry | Nistér 2004 | Five-point essential matrix solver, RANSAC, Triangulation, VO (local-only, no loop closure) |
| MonoSLAM | Davison 2007 | First real-time monocular SLAM, EKF-based, single camera, sparse 3D map, probabilistic feature initialization |
| PTAM | Klein & Murray 2007 | FAST feature, Tracking, Frontend/Backend separation, Parallel threads, Keyframe, Mapping, Bundle adjustment, Manual initialization |
| Visual-SLAM why filter? | Strasdat 2012 | Bundle adjustment, Scale-aware BA, Motion-only BA |
| ORB-SLAM | Mur-Artal 2015 | ORB keypoint, Automatic initialization (Homography vs Fundamental selection), Tracking thread, Local (covisibility-based) BA + global BA on loop closure, Local mapping, Large-scale, Loop closure, Bag of Visual Words, Global optimization, Covisibility graph, Map point management (culling, merging) |
| Pop-up SLAM | Yang 2016 | Line/Plane features |
| PL-SLAM | Pumarola 2017 | Point/Line features |
| ORB-SLAM2 | Mur-Artal 2017 | → Stereo SLAM, → RGB-D SLAM |
| CubeSLAM | Yang 2019 | Monocular 3D cuboid detection + SLAM, 9-DoF object representation |
| OpenVSLAM | Sumikura 2019 | ORB-based SLAM framework, perspective/fisheye/equirectangular camera models, map save/load + localization mode |
| Stella-VSLAM | Community 2021 | OpenVSLAM successor, license reboot (→ also in Level 7) |
| UcoSLAM | Muñoz-Salinas 2019 | Fiducial markers |
| DeepFusion | Laidlow 2019 | Dense monocular reconstruction, semi-dense MVS + CNN depth/gradient predictions, probabilistic fusion with learned uncertainties |
| ORB-SLAM3 | Campos 2020 | Monocular + Stereo + VIO, Multi-map, IMU integration |
| DXSLAM | Li 2020 | Deep features for SLAM |
| PyCuVSLAM | NVIDIA 2025 | Python + CUDA GPU-accelerated VSLAM toolkit (cuVSLAM wrapper; stereo/multi-camera VIO) |
Direct SLAM
| System | Author/Year | Key Concepts |
|---|---|---|
| DTAM | Newcombe 2011 | Dense mapping, Keyframe mapping, GPGPU |
| LSD-SLAM | Engel 2014 | Photometric error minimization, High gradient pixels/edges, Large scale, Loop closure, Pose graph optimization |
| DSO | Engel 2016 | Photometric bundle adjustment, Sliding window BA, No loop closure/global optimization |
| LDSO | Gao 2018 | DSO + Loop closure (BoW-based), addresses DSO's main weakness |
| CNN-SLAM | Tateno 2017 | Depth from LSD-SLAM + deep depth, Semantic label |
| DVSO | Yang 2018 | Deep single image depth estimation, StackNet |
| D3VO | Yang 2020 | Deep single image depth estimation, Deep pose, Deep aleatoric uncertainty |
Semi-direct (Hybrid)
| System | Author/Year | Key Concepts |
|---|---|---|
| SVO | Forster 2014 | FAST feature detection, sparse direct image alignment, depth filters |
| SVO2 | Forster 2017 | Multi-camera/Fisheye, Probabilistic depth estimation, Direct method convergence, Sparse method, Bundle adjustment |
| Stereo DSO | Wang 2017 | → Stereo SLAM |
| VI-DSO | von Stumberg 2018 | → VIO/VINS |
SfM Tools
| System | Author/Year | Key Concepts |
|---|---|---|
| COLMAP | Schönberger 2016 | De-facto standard incremental SfM + MVS pipeline (C++/CUDA, pycolmap bindings) |
| GLOMAP | Pan 2024 | Global SfM revisited — COLMAP-compatible, much faster mapping |
| InstantSfM | Zhong 2025 | GPU-native sparse-aware SfM pipeline, large speedups over COLMAP |
Dynamic Environment SLAM
| System | Author/Year | Key Concepts |
|---|---|---|
| DynaSLAM | Bescós 2018 | Mask R-CNN dynamic-object removal + background inpainting, ORB-SLAM2-based |
| DS-SLAM | Yu 2018 | Semantic segmentation (SegNet) + motion consistency check |
| MaskFusion | Rünz 2018 | RGB-D recognition, tracking and reconstruction of multiple moving objects |
| MID-Fusion | Xu 2019 | Octree-based object-level multi-instance dynamic RGB-D SLAM |
| VDO-SLAM | Zhang 2020 | Dynamic object-aware SLAM, joint camera + object motion estimation |
| DynaSLAM II | Bescós 2021 | Tightly-coupled multi-object tracking and SLAM |
| MonST3R | Zhang 2024 | DUSt3R-family pointmap estimation in the presence of motion |
Level 4: RGB-D Visual-SLAM
Key Concepts
- Depth from sensor — Structured light vs Active IR (ToF); metric scale for free, but range/material limitations
- Frame-to-model tracking — Aligning each frame against the accumulated model (ICP) instead of frame-to-frame
- TSDF vs Surfel maps — Volumetric signed-distance fusion (KinectFusion) vs point-based surfel fusion (ElasticFusion)
RGB-D Camera Devices
- Intel RealSense D series
- Orbbec Femto series (Azure Kinect successor), Orbbec Astra
- Luxonis OAK-D
- Legacy (discontinued): Microsoft Kinect v1/v2, Azure Kinect DK, Occipital Structure Core
GPGPU Programming
Systems
| System | Author/Year | Key Concepts |
|---|---|---|
| ICP | Besl & McKay 1992 | Iterative Closest Point, closest-point correspondence, closed-form rigid transform, local convergence (needs initialization), foundation of 3D-3D registration |
| DTAM | Newcombe 2011 | → see Level 3 Direct SLAM |
| KinectFusion | Newcombe 2011 | GPGPU, Tracking (project depth → 3D, surface normal, coarse-to-fine ICP), Mapping (volumetric integration, TSDF), Robust to small scene changes, Cannot model deformation, Map growth cubic, Room-size only |
| Double Window Optimisation | Strasdat 2011 | Inner window (local BA) + outer window (pose graph), covisibility graph, constant-time optimization |
| Kintinuous | Whelan 2012 | Volume shift, Geometric, Photometric, dBoW+SURF, Optimization, Loop closure |
| RGBD-SLAM-V2 | Endres 2013 | Tracking (color image, visual features, depth image, point cloud, transformation), Mapping (OctoMap 2013) |
| SLAM++ | Salas-Moreno 2013 | Object-oriented SLAM |
| DVO | Kerl 2013 | Keyframe, Depth, Direct method, Optimization, Loop closure |
| RTAB-Map | Labbé 2014 | Loop closure, Map merge, Multi-session memory management |
| MRS-Map | Stückler 2014 | Multi-resolution surfel maps in an octree, shape + color statistics per surfel, noise-aware RGB-D registration, real-time on CPU |
| ElasticFusion | Whelan 2015 | Active: frame-to-model tracking (photometric + geometric), joint optimization, fused surfel-based model reconstruction · Inactive: local loop closure (model-to-model local surface, submodel separation), global loop closure (randomised fern encoding, non-rigid space deformation) |
| DynamicFusion | Newcombe 2015 | 6D motion field, Deformable scene |
| ORB-SLAM2 (RGB-D mode) | Mur-Artal 2017 | Bundle adjustment, Sparse reconstruction (→ also in Level 3) |
| BundleFusion | Dai 2016 | Local-to-global optimization, Sparse RGB feature, Coarse global pose estimation, Fine pose refinement (geometric + photometric) |
| SemanticFusion | McCormac 2016 | Deep Learning CNN, Deep Semantic SLAM |
| InfiniTAM v3 | Prisacariu 2017 | Tracking (scene raycast, depth image, RGB image), Relocalization (random ferns), Mapping (TSDF reconstruction, voxel hashing, surfel reconstruction) |
| Fusion++ | McCormac & Clark 2018 | Deep Learning CNN, Mask-RCNN instance segmentation, Object-level SLAM, No prior, Object-level TSDF reconstruction |
| PointFusion / DenseFusion | Xu 2018 / Wang 2019 | RGB-D object 6-DoF pose estimation, point cloud + image feature fusion (object frontend for object-level SLAM) |
| BAD SLAM | Schöps 2019 | Direct RGB-D bundle adjustment, surfel map, real-time GPU BA, ETH3D benchmark |
| RTAB-Map (RGB-D / LiDAR) | Labbé 2019 | Multi-sensor RGB-D/LiDAR support, light-source detection (2016) |
| MoreFusion | Wada 2020 | DL instance segmentation, Object-level volumetric fusion, Volumetric pose prediction, 3D scene reconstruction, Collision-based refinement, Semantic SLAM, Object pose estimation, CAD object fitting |
| NodeSLAM | Sucar 2020 | Occupancy VAE, Object-level SLAM (→ also in Level 5 Latent Representation) |
| DSP-SLAM | Wang (UCL) 2021 | DeepSDF shape prior + ORB-SLAM2, object-level dense reconstruction (mono/stereo/LiDAR) |
Level 5: Applying Deep Learning
Key Concepts
- Learned vs hand-crafted — Replacing individual classical modules (features, depth, matching) with networks vs end-to-end learning
- Differentiability — Making classical optimization (RANSAC, BA) differentiable so it can be trained through
- Foundation models — Large pretrained models (CLIP, SAM, DUSt3R-family) as reusable perception backbones
Level 5 is organized into five pillars: A. Frontend — learned perception components replacing hand-crafted modules B. Backend — learned/certifiable optimization replacing classical solvers C. Systems — end-to-end deep VO/SLAM pipelines D. Scene Understanding — semantic, language, and relational reasoning on SLAM maps E. Foundation-Model & Neural SLAM — pointmap transformers, NeRF- and 3DGS-based dense SLAM systems
A. Deep Frontend — Perception
Feature Detection & Matching
| System | Author/Year | Key Concepts |
|---|---|---|
| NetVLAD | Arandjelović 2016 | VLAD, place recognition |
| SuperPoint | DeTone 2017 | Homographic Adaptation, Self-supervised, VGG encoder + detector/descriptor heads |
| HardNet | Mishchuk 2017 | Learned local descriptor |
| R2D2 | Revaud 2019 | Repeatable + Reliable detector/descriptor, explicit repeatability/reliability maps |
| KeyNet | Barroso-Laguna 2019 | Learned keypoint detector |
| HF-Net | Sarlin 2019 | Global feature, Local feature, Visual localization |
| SuperGlue | Sarlin 2020 | Self/Cross-attention GNN, Sinkhorn optimal assignment, dustbin for outliers |
| DISK | Tyszkiewicz 2020 | Policy gradient (RL) training, match success/failure as reward |
| Patch NetVLAD | Hausler 2021 | Multi-scale patch-level VLAD |
| LoFTR | Sun 2021 | Detector-free, Transformer coarse-to-fine dense matching |
| LightGlue | Lindenberger 2023 | Adaptive depth/width, 5-10× faster than SuperGlue |
| XFeat | Potje 2024 | 0.3M params, 1400 FPS (RTX 4090), 64-dim descriptor, embedded-friendly |
| RoMa | Edstedt 2024 | DINOv2 foundation feature + coarse-to-fine dense matching |
| DeDoDe | Edstedt 2024 | Joint detect-and-describe in one stage |
| RoMa v2 | Edstedt 2025 | Harder-better-faster-denser dense feature matching |
Depth Estimation
| System | Author/Year | Key Concepts |
|---|---|---|
| MonoDepth | Godard 2016 | Left-Right photometric consistency, self-supervised |
| MiDaS | Ranftl 2020 | Multi-dataset mixing, scale-and-shift invariant loss, relative depth |
| DPT | Ranftl 2021 | Dense Prediction Transformer (ViT backbone), global context |
| ZoeDepth | Bhat 2023 | Zero-shot metric depth, Metric Bins Module |
| Metric3D | Yin 2023 | Camera intrinsic-conditioned metric depth, Canonical Camera Space |
| Depth Anything | Yang 2024 | 62M images, foundation model for monocular depth |
| Depth Anything V2 | Yang 2024 | Improved with synthetic data, better edge preservation |
| Depth Anything 3 | Lin 2025 | Any-view geometry from arbitrary inputs, depth-ray prediction target, single plain transformer (DINOv2), teacher-student training |
| Marigold | Ke 2024 | Stable Diffusion for depth, fine detail, uncertainty via sampling |
| Align3R | Lu 2025 | Video temporal consistency, DUSt3R-based, CVPR 2025 Highlight |
| Masked Depth Modeling (LingBot-Depth) | Tan 2026 | Fixes RGB-D failures on glass/mirrors/metal |
Optical Flow & Scene Flow
| System | Author/Year | Key Concepts |
|---|---|---|
| FlowNet | Dosovitskiy 2015 | First end-to-end deep optical flow (SimpleNet / CorrNet) |
| FlowNet 2.0 | Ilg 2017 | Stacked networks, classical-level accuracy |
| PWC-Net | Sun 2018 | Pyramid-Warping-Cost volume, coarse-to-fine, 8.4M params |
| FlowNet3D | Liu 2019 | Point cloud scene flow, PointNet++ based |
| RAFT | Teed 2020 | All-Pairs Correlation + iterative ConvGRU update, ECCV Best Paper |
| RAFT-3D | Teed 2021 | Scene flow (3D motion) from RAFT |
| FlowFormer | Huang 2022 | Transformer on cost volume tokens, global context |
| SEA-RAFT | Wang 2024 | Efficient RAFT variant for real-time |
Camera Pose Regression & Relocalization
| System | Author/Year | Key Concepts |
|---|---|---|
| PoseNet | Kendall 2015 | CNN-based 6-DoF pose regression (APR), GoogLeNet backbone |
| DSAC | Brachmann 2017 | Differentiable RANSAC, Scene Coordinate Regression (SCR) |
| DSAC++ | Brachmann 2018 | Self-supervision, RGB-D support |
| CNN Pose Regression Limitations | Sattler 2019 | Pose regression ≈ image retrieval performance |
| LM-Reloc | von Stumberg 2020 | Deep direct relocalization |
| DSAC* | Brachmann 2021 | Visual relocalization from RGB/RGB-D, improved learning stability (TPAMI) |
| ACE | Brachmann 2023 | Accelerated Coordinate Encoding, 5-min training per scene |
| ACE Zero | Brachmann 2024 | Zero-shot SCR, no pre-built 3D map needed |
| ACE-G | Bruns 2025 | Generalizable SCR via query pretraining, new scenes without fine-tuning |
| ACE-SLAM | Alzugaray 2025 | Neural implicit real-time SLAM, network weights = map |
| hloc | Sarlin 2019 | Toolbox implementing HF-Net's hierarchical localization: coarse (NetVLAD) → fine (SuperGlue) |
Object Detection & Segmentation for SLAM
| System | Author/Year | Key Concepts |
|---|---|---|
| YOLO (v1→v11) | Redmon 2016→2024 | Real-time object detection, Ultralytics ecosystem |
| DETR | Carion 2020 | Transformer detection, anchor-free, no NMS |
| RT-DETR | Zhao (Baidu) 2023 | Real-time DETR, YOLO-speed + Transformer quality |
| RF-DETR | Robinson 2025 | Weight-sharing NAS over DETRs, accuracy-latency Pareto tuning, first real-time detector past 60 AP on COCO |
| SAM | Kirillov 2023 | Segment Anything, prompt-based, Foundation Model |
| SAM 2 | Meta 2024 | Video segmentation, Memory Attention, temporal consistency |
| SAM 3 | Carion 2025 | Promptable concept segmentation (noun-phrase / exemplar prompts), presence head, detector + memory-based video tracker |
| Grounding DINO | Liu 2023 | Text-prompted detection → SAM pipeline (Grounded SAM) |
| Open-YOLO 3D | Boudjoghra 2024 | 2D open-vocab detection → 3D instance seg, 16× faster |
B. Deep Backend — Optimization
Differentiable Bundle Adjustment
| System | Author/Year | Key Concepts |
|---|---|---|
| BA-Net | Tang 2019 | FPN + differentiable LM layer, end-to-end SfM (ICLR) |
| DROID-SLAM | Teed 2021 | Dense optical flow + differentiable dense BA, all-pixels reprojection |
| DPVO | Teed 2023 | Patch-based DROID-SLAM, 30+ FPS real-time |
| Theseus | Pineda (Meta) 2022 | Differentiable nonlinear optimization library (PyTorch) |
| Lietorch | Teed 2021 | Lie group operations for PyTorch (SE(3)/SO(3)) |
C. End-to-End Deep VO / SLAM Systems
Self-supervised & Learned VO
| System | Author/Year | Key Concepts |
|---|---|---|
| DeepVO | Wang 2017 | Supervised learning |
| SfM-Learner | Zhou 2017 | Unsupervised, deep depth + deep pose |
| DeMoN | Ummenhofer 2017 | Depth + Motion from two frames, encoder-decoder |
| UndeepVO | Li 2018 | Stereo self-supervised, absolute scale recovery |
| DeepTAM | Zhou 2018 | Deep tracking and mapping, cost volume based |
| DeepV2D | Teed 2018 | Iterative depth from video, differentiable geometry layers |
| Depth from Videos in the Wild | Gordon 2019 | Unconstrained video depth, learned camera intrinsics |
| Neural Ray Surfaces | Vasiljevic 2020 | Learned ray surface model, non-pinhole cameras |
| GradSLAM | Murthy 2020 | Differentiable SLAM framework (PyTorch, supports multiple SLAM backends) |
| DeepSLAM | Li 2020 | TrackingNet, MappingNet, LoopNet |
| MonoRec | Wimbauer 2021 | Self-supervised monocular 3D reconstruction, moving objects |
| TANDEM | Koestler 2021 | Real-time tracking + dense mapping via MVS depth, DSO-based |
Learning-based SLAM Systems
| System | Author/Year | Key Concepts |
|---|---|---|
| DROID-SLAM | Teed 2021 | Differentiable BA, dense optical flow, end-to-end learned |
| TartanVO | Wang 2021 | Generalizable visual odometry |
| DPV-SLAM | Lipson 2024 | DPVO + loop closure, full SLAM (ECCV 2024) |
| MAC-VO | Qiu 2024 | Learning-based VO, metric-aware |
| VoT | Yugay 2025 | Visual Odometry with Transformers (later retitled FVO) |
Latent Representation SLAM
| System | Author/Year | Key Concepts |
|---|---|---|
| CodeSLAM | Bloesch 2018 | Depth as 128-dim latent code, photometric BA on codes + poses |
| SceneCode | Zhi 2019 | Depth + semantic in single latent code, cross-modal constraints |
| DeepFactors | Czarnowski 2020 | Probabilistic depth codes + factor graph, GPU 30+ FPS |
| NodeSLAM | Sucar 2020 | Object-level DeepSDF codes, occupancy VAE per object |
| CodeMapping | Matsuki 2021 | Sparse SLAM + learned dense mapping, hybrid approach |
Neural Rendering (reference)
NeRF/3DGS-based SLAM systems → see Pillar E below
| System | Author/Year | Key Concepts |
|---|---|---|
| NeRF | Mildenhall 2020 | Neural Radiance Fields, novel view synthesis (foundational) |
| DIFIX3D+ | Wu 2025 | Single-step diffusion for 3D reconstruction artifact removal (post-processing) |
D. Scene Understanding
Benchmarks & Foundations
| System | Author/Year | Key Concepts |
|---|---|---|
| EFM3D | Straub (Meta) 2024 | Egocentric Foundation Model 3D benchmark, depth/surface/semantic from ego-video |
3D Scene Graph
| System | Author/Year | Key Concepts |
|---|---|---|
| Kimera / 3D Dynamic Scene Graph | Rosinol 2020 | Kimera-VIO, Kimera-Mesher, Kimera-PGMO, Kimera-Semantics, Kimera-DSG (stereo/mono visual-inertial pipeline) |
| Hydra | Hughes (MIT SPARK) 2022 | Real-time hierarchical Scene Graph (mesh→objects→places→rooms→buildings) |
| Hydra-Multi | Chang 2023 | Distributed multi-robot 3D Scene Graph |
| Clio | Maggio (MIT SPARK) 2024 | Open-set task-driven Scene Graph, CLIP embeddings per node |
| Khronos | Schmid (MIT SPARK) 2024 | Spatio-temporal Scene Graph, dynamic object history tracking |
| ConceptGraphs | Gu 2023 | Open-vocabulary 3D Scene Graph, SAM + CLIP + LLM relations |
Semantic / Language-Grounded SLAM
| System | Author/Year | Key Concepts |
|---|---|---|
| ConceptFusion | Jatavallabhula (MIT) 2023 | CLIP features fused into 3D map, open-vocabulary language queries |
| LERF | Kerr 2023 | Language Embedded Radiance Fields, DINO multi-scale, NeRF + CLIP |
| OpenScene | Peng (ETH) 2023 | Language features back-projected to 3D point clouds |
| SpatialLM | Mao 2025 | Point cloud → LLM, structured indoor modeling as Python scripts |
Also see: LEGS, OpenGS-SLAM (Pillar E above); Open-YOLO 3D (Level 5 Object Detection)
E. Foundation-Model & Neural-Representation SLAM
Foundation-Model SLAM
| System | Author/Year | Key Concepts |
|---|---|---|
| DUSt3R | Wang 2024 | Pointmap regression from image pairs, no calibration needed |
| MASt3R | Leroy 2024 | DUSt3R + local feature matching |
| MASt3R-SLAM | Murai 2024 | Real-time dense SLAM from MASt3R priors |
| VGGT | Wang (Meta) 2025 | Feed-forward inference of poses, depths, pointmaps, tracks from N views (CVPR 2025 Best Paper) |
| VGGT-SLAM | Maggio 2025 | Dense RGB SLAM optimized on the SL(4) manifold, VGGT frontend |
| VGGT-SLAM 2.0 | Maggio 2026 | Real-time dense feed-forward scene reconstruction |
| VGGT-Geo | Qin 2026 | Probabilistic geometric fusion of VGGT priors for dense indoor SLAM |
| IGGT | Li 2025 | Instance-grounded geometry transformer — unified 3D reconstruction + instance-level understanding |
| AMB3R | Wang 2025 | Accurate feed-forward metric-scale 3D reconstruction with backend, SfM/SLAM support |
| MASt3R-Fusion | Zhou 2025 | MASt3R feed-forward visual model + IMU + GNSS fusion |
NeRF-based
| System | Author/Year | Key Concepts |
|---|---|---|
| iMAP | Sucar 2021 | First NeRF-SLAM, single MLP, real-time tracking/mapping |
| BARF | Lin 2021 | Bundle-Adjusting NeRF, coarse-to-fine positional encoding, joint pose+NeRF opt (not full SLAM — pose+NeRF co-optimization) |
| NICE-SLAM | Zhu & Peng 2022 | Hierarchical feature grid (coarse/mid/fine), scalable |
| Co-SLAM | Wang 2023 | Hash grid (Instant-NGP) + coordinate encoding, 5-10× faster than NICE-SLAM |
| ESLAM | Johari 2023 | Tri-plane representation, O(N²) vs O(N³) memory |
| Point-SLAM | Sandström 2023 | Neural point cloud based |
| NeRF-SLAM | Rosinol 2023 | NeRF + classical SLAM pipeline |
| NICER-SLAM | Zhu 2024 | RGB-only NeRF-SLAM (no depth sensor), monocular depth integration |
| vMAP | Kong 2023 | Object-level NeRF-SLAM, per-object neural fields |
| GO-SLAM | Zhang 2023 | Global optimization + NeRF-SLAM, loop closure + global BA |
3DGS-based
| System | Author/Year | Key Concepts |
|---|---|---|
| SplaTAM | Keetha 2024 | Among the first 3DGS SLAM systems (concurrent with GS-SLAM, MonoGS), RGB-D, silhouette-guided densification |
| MonoGS | Matsuki 2024 | First monocular 3DGS SLAM (CVPR 2024 highlight), direct rasterization-based tracking, analytic camera Jacobians |
| GS-ICP SLAM | Ha 2024 | Gaussian-to-Gaussian ICP (Mahalanobis distance), geometric tracking |
| Photo-SLAM | Huang 2024 | Explicit geometry + implicit appearance (MLP color), anti-aliasing |
| RTG-SLAM | Peng 2024 | Real-time focus, adaptive Gaussian budget, Jetson Orin 25 FPS |
| EGG-Fusion | Pan 2025 | Geometry-aware Gaussian surfel fusion on the fly, information-filter-based, real-time |
| Online 3DGS Modeling | Lee 2025 | Online 3D Gaussian Splatting modeling with novel view selection |
| ActiveSplat | Li 2025 | Active mapping with 3DGS + Voronoi-based path planning |
| OpenGS-SLAM | Yang 2025 | Open-set dense semantic 3DGS SLAM, object-level scene understanding |
| LEGS | Yu 2024 | Language Embedded Gaussian Splats, real-time language-queryable 3D |
Level 6: VIO / VINS
Key Concepts
- Tightly-coupled vs Loosely-coupled — Joint vs separate optimization of visual and inertial measurements
- Filter-based vs Optimization-based — EKF approaches vs nonlinear optimization (BA)
- IMU preintegration — Integrating IMU measurements between keyframes (Lupton 2012; on-manifold formulation: Forster 2015)
- IMU noise model — Bias, random walk, Allan variance
- Observability — 4 unobservable DoF in VIO (3-DoF global translation + yaw); scale becomes additionally unobservable under constant-acceleration motion
- Deployed VIO — Commercial XR stacks (Meta Quest, ARKit/ARCore) are the highest-volume deployed VIO systems — worth studying as case studies
Foundations
| Resource | Author/Year | Key Concepts |
|---|---|---|
| Introduction to Inertial Navigation | Woodman 2007 | IMU fundamentals, coordinate frames, error sources — essential prerequisite |
| IMU Preintegration on Manifold | Forster 2015 | On-manifold preintegration, bias correction without re-integration |
| Quaternion kinematics for error-state KF | Solà 2017 | Quaternion math, error-state formulation |
Filter-based
| System | Author/Year | Key Concepts |
|---|---|---|
| MSCKF | Mourikis 2007 | Multi-State Constraint KF, efficient VIO without landmarks in state |
| ROVIO | Bloesch 2015 | Robocentric VIO, direct photometric tracking + EKF |
| OpenVINS | Geneva 2020 | Open-source MSCKF, modular, extensible |
Optimization-based
| System | Author/Year | Key Concepts |
|---|---|---|
| OKVIS | Leutenegger 2015 | Keyframe-based, tightly-coupled, sliding window optimization |
| VINS-Mono | Qin 2018 | Tightly-coupled, relocalization, loop closure, pose graph optimization |
| VI-DSO | von Stumberg 2018 | Direct sparse VIO, dynamic marginalization, photometric error |
| VINS-Fusion | Qin 2019 | Stereo + GPS fusion extension |
| maplab | Schneider 2018 | Multi-session visual-inertial mapping framework |
| Kimera-VIO | Rosinol 2020 | Fast VIO frontend for Kimera pipeline, structureless vision factors |
| Basalt | Usenko 2020 | Non-linear factor recovery (NFR) of marginalization priors, visual-inertial odometry + mapping |
| ORB-SLAM3 | Campos 2020 | VIO mode, multi-map, IMU initialization |
| DM-VIO | von Stumberg 2022 | Direct (DSO-based) monocular VIO, delayed marginalization, pose-graph BA for IMU initialization |
| OKVIS2 | Leutenegger 2022 | Multi-session, improved marginalization |
| AirVO | Xu 2023 | Point-line VIO, illumination-robust |
| OKVIS2-X | Boche & Leutenegger 2025 | Multi-sensor SLAM (Visual+Inertial+Depth+LiDAR+GNSS), dense volumetric occupancy maps, submapping for large-scale (9km+), EuRoC/Hilti22 SOTA |
Level 7: Stereo SLAM
Key Concepts
- Stereo rectification — Epipolar alignment for efficient disparity search
- Disparity vs Depth — d = f·B/Z, baseline determines depth range/accuracy
- Scale observability — Stereo provides metric scale (unlike monocular)
Systems
| System | Author/Year | Key Concepts |
|---|---|---|
| S-PTAM | Pire 2017 | Stereo PTAM, ROS-compatible, real-time |
| ORB-SLAM2 (stereo) | Mur-Artal 2017 | Stereo + RGB-D modes, loop closure, relocalization |
| StereoMSCKF | Sun 2018 | MSCKF with stereo, efficient for resource-constrained platforms |
| RTAB-Map | Labbé 2019 | Multi-sensor (stereo/RGB-D/LiDAR), memory management, large-scale (→ also in Level 4) |
| ORB-SLAM3 (stereo) | Campos 2020 | Multi-map, Atlas, stereo + IMU |
| Stella-VSLAM | Community 2021 | OpenVSLAM successor, stereo support (→ also in Level 3) |
| Stereo DSO | Wang 2017 | Direct sparse stereo odometry, large-scale (DSO extension) |
Level 8: Collaborative / Multi-Robot SLAM
Key Concepts
- Centralized vs Decentralized — Single server vs peer-to-peer map merging
- Inter-robot loop closure — Place recognition across robots with different viewpoints
- Communication constraints — Bandwidth-limited map sharing, sparse descriptors
- Map merging — Aligning submaps from different robots into a global map
Systems
| System | Author/Year | Key Concepts |
|---|---|---|
| C2TAM | Riazuelo 2014 | Cloud-based collaborative monocular SLAM |
| CCM-SLAM | Schmuck & Chli 2019 | Centralized collaborative monocular SLAM, robust to comm failures |
| DOOR-SLAM | Lajoie 2020 | Distributed, outlier-resilient SLAM with pairwise consistency |
| Kimera-Multi | Tian 2022 | Distributed multi-robot metric-semantic SLAM, mesh reconstruction |
| Swarm-SLAM | Lajoie 2024 | Decentralized, sparse, scalable C-SLAM, supports LiDAR/stereo/RGB-D |
| CoPeD | Zhou 2024 | Multi-robot collaborative perception dataset (real-world, aerial + ground robots) |
| maplab 2.0 | Cramariuc 2023 | Multi-session, multi-robot visual-inertial mapping |
Level 9: LiDAR & Visual-LiDAR Fusion SLAM
Key Concepts
- LiDAR-Visual-Inertial (LVI) — Triple fusion for robust outdoor SLAM
- Tightly-coupled LiDAR-camera — Joint optimization of point cloud and visual features
- Direct LiDAR-camera alignment — Photometric/geometric alignment without feature extraction
- Degradation handling — Graceful fallback when one modality fails (e.g., LiDAR in rain, camera in darkness)
- Range image — 2D projection of LiDAR scans for efficient processing (SuMa, RangeNet++)
LiDAR / LiDAR-Inertial SLAM
| System | Author/Year | Key Concepts |
|---|---|---|
| LOAM | Zhang 2014 | LiDAR odometry and mapping (foundational), edge + planar features |
| SuMa | Behley (Bonn) 2018 | Surfel-based LiDAR SLAM, projective ICP on range images |
| SuMa++ | Chen (Bonn) 2019 | SuMa + RangeNet++ semantics, semantic ICP weighting, dynamic object filtering |
| LIO-SAM | Shan 2020 | Tightly-coupled LiDAR-inertial, factor graph, GPS fusion |
| FAST-LIO2 | Xu 2022 | Direct LiDAR-inertial, ikd-Tree, extremely fast |
| PIN-SLAM | Pan (Bonn) 2024 | Neural point cloud LiDAR SLAM, point-to-SDF registration, elastic map deformation for loop closure |
Visual-LiDAR Fusion SLAM
| System | Author/Year | Key Concepts |
|---|---|---|
| LVI-SAM | Shan 2021 | LiDAR-Visual-Inertial via factor graph, LIO-SAM + VINS-Mono |
| R3LIVE | Lin 2022 | Real-time LiDAR-Visual-Inertial, dense RGB point cloud map |
| R3LIVE++ | Lin 2023 | Improved R3LIVE with mesh reconstruction |
| FAST-LIVO | Zheng 2022 | FAST-LIO + direct visual odometry, tightly-coupled LVI |
| FAST-LIVO2 | Zheng 2024 | Improved, sequential image processing, direct photometric fusion |
| OKVIS2-X | Boche 2025 | Visual+Inertial+Depth+LiDAR+GNSS configurable (also in Level 6) |
Resources
| Resource | Author/Year | Key Concepts |
|---|---|---|
| Multi-Sensor Fusion SLAM Survey | Zhu 2024 | Camera + LiDAR + IMU fusion SLAM — comprehensive survey |
Level 10: Event Camera SLAM
Key Concepts
- Event cameras (DVS) — Asynchronous per-pixel brightness change detection, μs temporal resolution
- Advantages — HDR (140dB+), no motion blur, low latency, low power
- Challenges — No absolute intensity, sparse asynchronous output, requires new algorithms
- Event representations — Event frames, time surfaces, voxel grids, spike tensors
Foundations
| Resource | Author/Year | Key Concepts |
|---|---|---|
| Event-based Vision Survey | Gallego 2020 | Comprehensive survey of event camera algorithms |
| Awesome-Event-based-SLAM | KwanWaiPang | Curated GitHub list of event-based SLAM papers |
Systems
| System | Author/Year | Key Concepts |
|---|---|---|
| EVO | Rebecq 2017 | Event-based Visual Odometry, 3D reconstruction from events |
| ESVO | Zhou 2021 | Event-based Stereo Visual Odometry |
| Ultimate-SLAM | Vidal 2018 | Events + frames + IMU fusion |
| EKLT | Gehrig 2020 | Event-based KLT feature tracking |
| ESVIO | Chen 2023 | Event-based Stereo VIO |
| EDS | Hidalgo-Carrió 2022 | Event-aided direct sparse odometry |
| DEVO | Klenk 2024 | Deep event-based visual odometry, DPVO-style patch-based, trained on simulated events |
| VIO-GO | Sakhrieh 2025 | Event-based VIO with optimized parameters for HDR scenarios |
Level 11: World Models & Spatial AI
Key Concepts
- World model — A learned generative model of environment dynamics, usable for prediction and planning
- VLM vs VLA — Vision-language models reason about images; vision-language-action models additionally output robot actions
- Spatial AI — The convergence of SLAM, scene understanding, and learned world representations (Davison's FutureMapping vision)
World Models
| System | Author/Year | Key Concepts |
|---|---|---|
| GAIA-1 | Wayve 2023 | Driving World Model, action-conditioned future scene generation |
| Sora / DiT | OpenAI 2024 | Diffusion Transformer, spacetime patches, emergent 3D understanding |
| NVIDIA Cosmos | NVIDIA 2025 | World Foundation Model platform for Physical AI, synthetic data for AV/robots |
| World Labs / Marble | Fei-Fei Li 2025 | Generative 3D worlds (persistent Gaussian-splat scenes) from image/video/text prompts |
| WorldVLA | Cen (Alibaba) 2025 | Autoregressive action world model, learns physics for action generation |
| SceneDINO | Jevtić 2025 | Feed-forward unsupervised semantic scene completion |
Generative 3D
| System | Author/Year | Key Concepts |
|---|---|---|
| DreamFusion | Poole 2023 | Text-to-3D via Score Distillation Sampling (SDS) + NeRF |
Vision-Language Models (VLM)
| System | Author/Year | Key Concepts |
|---|---|---|
| CLIP | Radford (OpenAI) 2021 | Contrastive image-text pretraining, 400M pairs, zero-shot |
| SigLIP | Zhai (Google) 2023 | Sigmoid loss CLIP, more efficient, better at small model sizes |
| BLIP-2 | Li (Salesforce) 2023 | Q-Former bridges frozen LLM + image encoder |
| LLaVA | Liu 2023 | LLaMA + vision, conversational VLM |
Vision-Language-Action Models (VLA)
| System | Author/Year | Key Concepts |
|---|---|---|
| RT-2 | Brohan (DeepMind) 2023 | Robot actions as text tokens, emergent generalization |
| OpenVLA | Kim 2024 | Open-source VLA, SigLIP + Llama 7B + Action Head |
| NaVILA | Cheng 2024 | Legged/wheeled-robot vision-language-action model for navigation |
Resources
| Resource | Author/Year | Key Concepts |
|---|---|---|
| Awesome-Transformer-based-SLAM | KwanWaiPang | Curated GitHub list of Transformer-based SLAM methods |
Study Resources
YouTube Lecture Series
| Lecture | Instructor | Link |
|---|---|---|
| SLAM & Photogrammetry | Cyrill Stachniss (Uni Bonn) | YouTube Playlist |
| First Principles of Computer Vision | Shree Nayar (Columbia) | YouTube Channel |
| Multiple View Geometry | Daniel Cremers (TU Munich) | YouTube Playlist |
Books
| Book | Author | Key Topics |
|---|---|---|
| Introduction to Visual SLAM | Xiang Gao et al. | VO, optimization, Lie algebra, backend, loop closure — best entry-level SLAM book |
| Photogrammetric Computer Vision | Wolfgang Förstner & Bernhard Wrobel | Camera geometry, estimation, 3D reconstruction — mathematically rigorous |
| Multiple View Geometry in Computer Vision | Richard Hartley & Andrew Zisserman | Epipolar geometry, trifocal tensor, reconstruction — THE bible |
| Computer Vision: Algorithms and Applications | Richard Szeliski | Feature detection, stereo, motion, 3D — comprehensive reference (2nd ed. free PDF) |
| State Estimation for Robotics | Timothy Barfoot | Estimation theory, Lie groups, batch/recursive estimation — free PDF (2nd ed.) |
| Probabilistic Robotics | Thrun, Burgard & Fox | Bayes filters, EKF/particle-filter SLAM — the classical probabilistic foundation |
| Factor Graphs for Robot Perception | Frank Dellaert & Michael Kaess | Factor graphs, elimination, iSAM2 — the backend bible (free PDF) |
| SLAM Handbook | Carlone, Kim, Barfoot, Cremers, Dellaert (eds.) | From localization and mapping to spatial intelligence — free community book (2024-25) |
Surveys
| Survey | Author/Year | Key Concepts |
|---|---|---|
| Past, Present, and Future of SLAM | Cadena 2016 | The canonical orientation survey — robust perception age, open problems |
| Event-based Vision Survey | Gallego 2020 | Event cameras and algorithms (→ also in Level 10) |
Code & Practice
| Resource | Link |
|---|---|
| SLAM Zero-to-Hero code exercises | GitHub — Docker-based hands-on exercises for this roadmap's topics (feature detection, epipolar geometry, RANSAC, ICP, g2o/GTSAM/Ceres) and systems (ORB-SLAM2, Basalt, Kimera, FAST-LIO2, MASt3R-SLAM, ...); individual exercises are linked from the matching study notes |
| changh95/slam_lecture_codes | GitHub — Full SLAM lecture code collection |
Wrap Up
If you think any part of the roadmap can be improved, please open a PR or submit an issue. I will continue to improve this, so you may want to watch/star this repository and revisit it later.
Contribution
- Open pull request with improvements
- Discuss ideas in issues
- Spread the word
- Reach out to me directly at hyunggi.chang95[at]gmail.com
Discussion
To discuss any topics or ask questions, please use the issue tab.
License
This project is licensed under the MIT License:
Copyright © 2026 Hyunggi Chang.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
awesome
awesome-list
computer-vision
deep-learning
rgb-d
roadmap
robotics
slam
vio
visual-inertial-odometry
visual-slam
Similar repos
Contributors
changh95
59 commits
Languages
Astro
69.3%
JavaScript
22.8%
CSS
6.0%
TypeScript
1.9%