@misc{shi2026show,
      title={Scene and Human in One World: Reconstruction in a Feedforward Pass},
      author={Boao Shi and Qiao Feng and Yiming Huang and Lingjie Liu},
      year={2026},
      eprint={2606.27720},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2606.27720},
}

[!IMPORTANT] Repository Status: Under Construction

The author is currently prioritizing postgraduate applications. The provided raw code is reproducible, but comprehensive documentation and cleanup are pending.

Clean-up will commence after the application period ends. Thank you for your interest and support!

SHOW reconstructs the full 3D world — the scene (cameras, depth maps, point maps) and the human (SMPL-X body mesh) — from egocentric video in a single feedforward pass. It builds on VGGT (CVPR 2025 Best Paper) and extends it with (i) a DensePose-pretrained backbone that injects projected depth features into a human-mask token decoder, and (ii) a promptable SMPL-X decoder that regresses the human in a unified world coordinate system with the scene.

This repository contains the full training, inference, and evaluation code to reproduce the paper.

---

Table of Contents

• 1. Repository Structure
• 2. Installation
• 3. Download Checkpoints
• 4. Prepare Datasets
• 5. Training
• 6. Inference
• 7. Evaluation
• 8. Reproducibility Notes & FAQ
• 9. License & Acknowledgements

---

1. Repository Structure

VGGT-HMR/
├── training/                     # Multi-stage training framework
│   ├── launch.py                 #   Training entry point: torchrun launch.py --config <name>.yaml
│   ├── trainer.py                #   Trainer (DDP, AMP, resume, logging)
│   ├── loss.py                   #   Multitask loss (camera/depth/point/densepose/smplx/...)
│   ├── config/                   #   Hydra configs: one yaml per experiment/stage
│   ├── data/                     #   Dataset classes: bedlam2, co3d, humman, behave, bedlam,
│   │                             #   scannet, tartanair, vkitti + composed/dynamic dataloaders
│   └── logs/                     #   (gitignored) checkpoints & tensorboard per experiment
├── inference/
│   ├── inference.py              # Video inference entry point
│   ├── inference_util.py         # load_model, inference_sequence, visualization
│   └── utils/                    # data/smpl utils shared with training
├── evaluation_tools/
│   ├── evaluate.py               #   Evaluate SHOW (VGGT-HMR) on 3DPW / EMDB / RICH
│   ├── evaluate_unish.py         #   Evaluate the UniSH baseline on the same benchmarks
│   ├── segmentation.py           #   SAM3-based human-mask preprocessing for eval
│   └── eval_results/             #   (gitignored) per-sequence metrics + progress JSON
├── eval/                         # Human3R baseline harness (relpose TUM / video_depth BONN / global_human)
├── unish/                        # UniSH baseline model code (Pi3 + human head + alignnet)
├── hmr4d/                        # GVHMR-derived dataset readers & mocap metrics for evaluation
├── vggt/                         # VGGT backbone + new SHOW modules (mask encoder, human mask
│                                 # depth token decoder, geometry injector, SMPL-X decoder)
├── densepose/                    # DensePose supervision utilities (smplx_densepose.npz)
├── ckpts/                        # Auxiliary checkpoints (e.g. yolo11n.pt for human detection)
└── scripts/                      # fetch_smplx.sh, fetch_bedlam.sh, fetch_model.sh, ...

---

2. Installation

Requirements. Linux + NVIDIA GPU (Ampere or newer recommended; training uses bf16 AMP). We used PyTorch 2.3.1 / torchvision 0.18.1 with CUDA 12.x, 4×A100/H100 80GB for the paper's runs (configs also run on fewer/smaller GPUs, see FAQ).

# 1. Clone
git clone <your-repo-url> VGGT-HMR
cd VGGT-HMR

# 2. Python env (conda example)
conda create -n show python=3.10 -y
conda activate show
pip install torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu121
pip install numpy==1.26.1 Pillow huggingface_hub einops safetensors opencv-python tqdm

# 3. Base package (VGGT) + training & eval dependencies
pip install -e .
pip install hydra-core omegaconf fvcore wandb tensorboard
pip install trimesh open3d scipy roma pyrender
pip install "ultralytics"          # YOLO human detector (inference)
pip install "segment-anything-2"    # SAM2 masks (inference); "sam3" also needed for eval masks
pip install smplx                   # SMPL-X body model
pip install accelerate              # baseline eval harness (eval/)

Always run from the repository root, with the root on PYTHONPATH:

export PYTHONPATH=$(pwd):$PYTHONPATH

---

3. Download Checkpoints

The following table lists every checkpoint needed for training, inference, or evaluation. Set the paths in the configs as described below (all configs take absolute paths).

⚠️ SHOW released checkpoints (placeholder). The final model checkpoints will be released on Hugging Face (linked from the project page). Expected files:

<HF_REPO>/model.pt                   # final model (exp_46 full-params finetune)
<HF_REPO>/checkpoints/exp_44_pretrain.pt
<HF_REPO>/checkpoints/exp_45_finetune.pt

Download them (e.g. huggingface-cli download <HF_REPO> --local-dir checkpoints/) and point resume_checkpoint_path / --checkpoint at the local files.

3.1 VGGT-1B (start point for Stage 1)

mkdir -p pretrained_model/vggt
wget -O pretrained_model/vggt/model.pt \
  https://huggingface.co/facebook/VGGT-1B/resolve/main/model.pt

3.2 SMPL-X body models

Register at https://smpl-x.is.tue.mpg.de, then:

bash scripts/fetch_smplx.sh        # prompts for your credentials

The configs reference the models_lockedhead variant used for training, e.g.:

model_folder: "/path/to/smplx_neutral_head/models_lockedhead/smplx"

3.3 PromptHMR checkpoint

mkdir -p data/pretrain/phmr
# download "checkpoint.ckpt" from https://github.com/microsoft/PromptHMR (data/pretrain/phmr/checkpoint.ckpt)
# or from the UniSH repo (https://huggingface.co/Murphyyyy/UniSH)

Set in every training config:

model:
  cfg:
    smplx_cfg:
      PHMR_PRETRAINED_CKPT_PATH: "/path/to/data/pretrain/phmr/checkpoint.ckpt"

---

4. Prepare Datasets

4.1 Training data — BEDLAM2

The paper's training pipeline uses BEDLAM (rendered egocentric-style videos with SMPL-X ground truth). Layout expected by training/data/datasets/bedlam2.py:

BEDLAM2_DIR/
└── <category>/                        # e.g. agora, synthetic, ...
    ├── png/<seq>/<seq>_<view_index>.png
    ├── exr_depth/<seq>/<seq>_<view_index>.exr
    └── exr_layers/masks/<seq>/<seq>_<view>_<human_id>_{hair,body,clothing}.png

BEDLAM2_LABEL/
└── <category>/<seq>.npz               # per-sequence SMPL-X annotations

Steps:

1. Register at https://bedlam.is.tue.mpg.de/ and download the image/depth/exr_layers data for the categories you use.

2. Download the label files:

   bash scripts/fetch_bedlam.sh        # downloads all_npz_12_training.zip & _validation.zip
   

   Unzip into BEDLAM2_LABEL so that each <category>/<seq>.npz sits under the same category name used in BEDLAM2_DIR.

3. Point both paths in every training config:

   data:
     train:
       dataset:
         dataset_configs:
           - _target_: data.datasets.bedlam2.BEDLAM2Dataset
             split: train
             BEDLAM2_DIR: /path/to/bedlam2
             BEDLAM2_LABEL: /path/to/bedlam2_labels
   

Other supported datasets (swappable in data.train.dataset.dataset_configs, all implemented in training/data/datasets/): Co3dDataset, HummanDataset, BehaveDataset, BEDLAMDataset, ScannetDataset, TarTanAirDataset, VKittiDataset. Mix datasets with ComposedDataset; the sampling ratio is controlled by each dataset's len_train.

4.2 Evaluation data — 3DPW / EMDB / RICH

The evaluation harness (evaluation_tools/) reads datasets through the vendored hmr4d package. Expected layout:

eval_dataset/
├── 3DPW/
│   └── imageFiles/<seq_name>/image_00000.jpg ...     # 3DPW test sequences
├── EMDB/
│   └── <P_number>/<seq_name>/images/...              # EMDB split 1 & 2
└── RICH/
    └── <seq_name>/...                                # RICH images

Prerequisite (GVHMR support files). The hmr4d dataset readers additionally need the GVHMR preprocessed files under inputs/ (from the GVHMR repo, data/ folder):

inputs/
├── 3DPW/hmr4d_support/     test_3dpw_gt_labels.pt, preproc_test_bbx.pt, preproc_test_kp2d_v0.pt
├── EMDB/hmr4d_support/     emdb_vit_v4.pt, emdb_dpvo_traj.pt
└── RICH/hmr4d_support/     ...

⚠️ Hardcoded paths. hmr4d/dataset/{threedpw,emdb,rich}/*_motion_test.py currently hardcode the eval-data root (e.g. /vast/projects/.../eval_dataset/3DPW). Change self.dataset_root in those three files to your local paths before evaluating.

Human masks. Evaluation uses per-frame human segmentation masks (masks_sam3/ for 3DPW, masks/ for EMDB, derived from the image paths by string replacement). Masks are generated with SAM3 via evaluation_tools/segmentation.py (run once per dataset before evaluating; see comments in evaluate.py).

---

5. Training

5.1 Overview — the three-stage pipeline

Training is organized as three sequential stages, each with its own config. Each stage resumes from the checkpoint produced by the previous stage via checkpoint.resume_checkpoint_path (strict: False, so partial/architecture-mismatched weights load cleanly). Training runs from training/:

cd training
torchrun --nproc_per_node=<N> launch.py --config <exp_name>.yaml

The configs in cmd.txt also document earlier intermediate stages (exp_42 mask pretrain → exp_43 geometry token) that led to the final exp_44 recipe.

5.2 Shared config edits (required)

Every config in training/config/ has hardcoded machine-specific paths. Before training, edit these fields (they appear at the top of each yaml):

model_folder:            "/path/to/.../smplx_neutral_head/models_lockedhead/smplx"   # SMPL-X bodies
densepose_rgb_path:      "/path/to/VGGT-HMR/densepose/smplx_densepose.npz"
data:                    # BEDLAM2_DIR / BEDLAM2_LABEL in train & val dataset_configs
checkpoint:
  resume_checkpoint_path: "/path/to/previous-stage/checkpoint_N.pt"   # see stage table
model:
  cfg:
    smplx_cfg:
      PHMR_PRETRAINED_CKPT_PATH: "/path/to/phmr/checkpoint.ckpt"

Checkpoints are saved to training/logs/<exp_name>/ckpts/checkpoint_<epoch>.pt (checkpoint.save_freq: 5 epochs) plus a checkpoint.pt "latest" copy. TensorBoard logs go to training/logs/<exp_name>/tensorboard.

5.3 Stage 1 — Pretrain the backbone (DensePose + projected depth + mask tokens)

cd training
torchrun --nproc_per_node=4 launch.py --config exp_44_pretrain_with_projected_depth_dpt_VGGT_feature.yaml

Key settings (exp_44_pretrain...yaml):

• model.cfg.stage: "pretrain", enable_densepose: True, enable_vggt_mask_encoder: True, enable_smplx: False
• Losses: camera (w=5), depth (w=1), point (w=1), mask-supervision (w=1), densepose (w=1)
• optim.frozen_module_names: ["*camera_head*"] — backbone is trained, camera head frozen
• max_img_per_gpu: 40, img_size: 518, patch_size: 14, lr: 5e-5 (AdamW, wd 0.05, cosine decay), bf16 AMP
• limit_train_batches: 80000, limit_val_batches: 4000

Note: the shipped configs' resume_checkpoint_path entries are continuation paths from the authors' own runs (e.g. an experiment resuming its own checkpoint_135.pt). For a clean reproduction, point each stage's resume_checkpoint_path at the previous stage's checkpoint (VGGT-1B model.pt for Stage 1) — loading is strict: False and works across architecture changes between stages.

5.4 Stage 2 — Finetune the SMPL-X decoder

cd training
# 2a. first decoder finetune (resumes from Stage-1 checkpoint_80.pt — set resume_checkpoint_path in the config)
torchrun --nproc_per_node=4 launch.py --config exp_44_finetune_smpl_decoder_after_pretrain_with_projected_depth_dpt_VGGT_feature.yaml

# 2b. continue with the 17-feature DPT recipe (resumes from 2a's checkpoint, e.g. checkpoint_155.pt)
torchrun --nproc_per_node=4 launch.py --config exp_45_finetune_smpl_decoder_after_pretrain_with_17_projected_depth_dpt_VGGT_feature.yaml

Key settings:

• model.cfg.enable_smplx: True — SMPL-X decoder + PHMR image encoder + prompt encoder are added on top of the VGGT backbone
• Losses add smplx_param, scale, joints_3d, vertices_3d, joints_2d, smplx_transl
• Frozen: camera head, image encoder, prompt encoder, cam encoder (check frozen_module_names)
• Resumes from the Stage-1 output (exp_44_pretrain.../checkpoint_80.pt in the reference runs — set checkpoint.resume_checkpoint_path to your own Stage-1 checkpoint_N.pt)

5.5 Stage 3 — Full fine-tuning (final model)

cd training
torchrun --nproc_per_node=4 launch.py --config exp_46_finetune_smpl_decoder_after_pretrain_with_17_projected_depth_dpt_VGGT_feature_full_params.yaml

Key settings: same architecture as Stage 2, but all modules are trainable except camera_head/image_encoder/prompt_encoder/cam_encoder as configured — this is the setting used for the numbers reported in the paper. Resumes from the Stage-2 checkpoint (e.g. exp_45.../checkpoint_300.pt).

5.6 Ablations

The ablations from the paper are provided as ready-to-run configs:

Run them exactly like the stages above:

torchrun --nproc_per_node=2 launch.py --config ablation_2_pretrain_vggt_raw_finetune_decoder.yaml

---

6. Inference

Inference takes a video file (or a folder of frames), detects the human (YOLO + SAM2 masks), runs the model in sliding chunks, and outputs the scene point cloud, human SMPL-X meshes, per-frame cameras, and a rendered video.

python inference/inference.py \
    --video_path path/to/video.mp4 \                    # or a folder of images
    --checkpoint  checkpoints/model.pt \                # SHOW final checkpoint
    --cfg         training/config/exp_46_finetune_smpl_decoder_after_pretrain_with_17_projected_depth_dpt_VGGT_feature_full_params.yaml \
    --output_dir  inference_results_video \
    --body_models_path /path/to/body_models             # SMPL/SMPL-X root (PromptHMR-style layout)

Example from cmd.txt:

python inference/inference.py \
    --video_path test_imgs/downtown_bar_00 \
    --checkpoint training/logs/exp_45_finetune_smpl_decoder_after_pretrain_with_17_projected_depth_dpt_VGGT_feature/ckpts/checkpoint_300.pt \
    --cfg training/config/exp_45_finetune_smpl_decoder_after_pretrain_with_17_projected_depth_dpt_VGGT_feature.yaml

Important options

Outputs (in --output_dir)

<seq_name>/
├── <seq_name>.mp4                  # rendered visualization
├── smpl_meshes/                    # SMPL-X mesh .ply per frame
├── scene_point_clouds/             # scene-only (human removed) point clouds
├── human_point_clouds/             # human point clouds
└── camera_parameters/              # per-frame camera parameters

Running the UniSH baseline (same CLI, model auto-downloaded from Hugging Face):

python inference/inference.py --video_path path/to/video.mp4 --model_type unish

---

7. Evaluation

Two quantitative benchmarks are used for the human reconstruction results: 3DPW (test), EMDB (split 1 and 2), and RICH. Metrics are the standard mocap metrics from GVHMR's ThreeDPWMetricMocap / EMDBMetricMocap / RichMetricMocap (PA-MPJPE, MPJPE, etc.). The eval/ folder additionally contains the Human3R baseline harness for relative pose (TUM), video depth (BONN), and global human motion.

7.1 Evaluate SHOW (our model)

python evaluation_tools/evaluate.py \
    --eval_dataset 3dpw \                          # or emdb1 | emdb2 | rich
    --model_ckpt_path checkpoints/model.pt \
    --model_cfg_path training/config/exp_46_finetune_smpl_decoder_after_pretrain_with_17_projected_depth_dpt_VGGT_feature_full_params.yaml

• --downsample_rate controls temporal subsampling (default 1 = full frame rate).
• Results are written to evaluation_tools/eval_results/<dataset>/; a *_progress.json records per-sequence metrics so evaluation resumes automatically after an interruption (already-completed sequences are skipped).
• Prerequisite: eval datasets + GVHMR inputs/ support files + SAM3 masks (see §4.2).

7.2 Evaluate the UniSH baseline

python evaluation_tools/evaluate_unish.py

The UniSH checkpoint (unish_release.safetensors) is downloaded automatically from Hugging Face (Murphyyyy/UniSH) on first run; results land in evaluation_tools/eval_results/unish/. Note: this script currently hardcodes the dataset (3dpw) and saves into a unish/ result dir — edit the eval_dataset variable / checkpoint path at the bottom of the file to run other splits.

7.3 Baseline evaluation harness (Human3R)

The eval/ folder reproduces the Human3R baselines from the paper:

These scripts use accelerate launch and expect the Human3R weights at the path in each run.sh (model_weights=...; edit it to your human3r_896L.pth).

---

8. Reproducibility Notes & FAQ

OOM during training. Reduce max_img_per_gpu and/or set accum_steps: 2 (gradient accumulation) in the config. Gradient clipping is already configured per-module (GradientClipper).

Learning rate. The effective batch size is max_img_per_gpu × num_gpus. If you change the number of GPUs / batch size, tune lr accordingly — try 5e-6, 1e-5, 5e-5, 1e-4, 5e-5 is what the paper uses.

Seed. seed_value: 42 is set per config; inference uses --seed 42.

Coordinate conventions. Cameras follow OpenCV camera-from-world; depth is aligned to its camera pose; SMPL-X translations are in the world frame of the scene (meters).

Where do checkpoints live? training/logs/<exp_name>/ckpts/checkpoint_<epoch>.pt (latest copy: checkpoint.pt). Loading uses strict: False, so you can resume across stages even if heads were added/removed.

Logging. TensorBoard under training/logs/<exp_name>/tensorboard; the trainer also logs scalar keys listed in logging.scalar_keys_to_log (per-loss breakdown).

Known loose ends for open-source reproduction (help wanted / to be cleaned up):

• Eval dataset roots are hardcoded in hmr4d/dataset/*/*_motion_test.py; make them configurable (env var) before releasing.
• evaluate_unish.py's main is hardcoded; refactor to the same argparse style as evaluate.py.
• Some auxiliary checkpoints (PromptHMR checkpoint.ckpt, GVHMR inputs/ support files) must be fetched from their upstream repos.
• Hardware: the reference runs in cmd.txt use 2–8 GPUs (mostly 4). If you have fewer/smaller GPUs, lower max_img_per_gpu and (if needed) enable accum_steps — expected results are the same, just slower.

---

9. License & Acknowledgements

See LICENSE for the license of this repository. This project builds on VGGT (CVPR 2025), GVHMR, PromptHMR, UniSH, Human3R, BEDLAM, and many other great open-source projects. If you use this code, please cite:

@misc{shi2026show,
      title={Scene and Human in One World: Reconstruction in a Feedforward Pass},
      author={Boao Shi and Qiao Feng and Yiming Huang and Lingjie Liu},
      year={2026},
      eprint={2606.27720},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2606.27720},
}