TL;DR — A learned HSI prior that predicts iterative geometry-grounded refinements to reconstruct humans in scenes from a single image. Matches optimization-based contact quality at ~100× lower runtime; as a plug-in prior it boosts existing methods by up to 44% contact F1 — no retraining needed.

https://github.com/user-attachments/assets/0ce2b017-afa6-49c5-a016-d007e1ef7a93

This repository is the code release: the model, the demo, and the harness that reproduces the paper's PROX and RICH numbers.

from graft import GRAFTModel

model = GRAFTModel.from_pretrained("graft").cuda().eval()
out = model.predict("photo.jpg", scene_image="empty_room.jpg")

out.vertices     # (n_people, 10475, 3) SMPL-X vertices, scene frame, metres
out.smplx        # dict of SMPL-X parameters
out.points       # (H, W, 3) metric scene point map

GRAFT refines an existing per-frame pose estimate rather than detecting people itself, so a photo with nobody in it raises graft.NoPeopleDetected.

---

Roadmap

• Project page
• arXiv paper
• Demo / inference code
• Evaluation code and the PROX / RICH benchmark harness
• Training code

---

Install

conda create -n graft python=3.10 && conda activate graft
bash install.sh                           # everything (--minimal for inference only)
python scripts/download_assets.py --eval  # weights and assets (drop --eval to skip benchmarks)
python scripts/check_install.py --all     # confirm the environment is ready

install.sh installs torch from the CUDA-matched index (cu124; override with TORCH_INDEX), then the project and its extras one at a time, so a failing extra still leaves a working inference install. requirements-lock.txt records the versions the published numbers were produced with.

The download pulls ≈1.1 GB into $GRAFT_DATA_ROOT (default ./data): our weights and benchmark assets from ympradyumna/graft, the NLF detector from its GitHub release, and the SMPL-X body models from an account-gated portal — register at https://smpl-x.is.tue.mpg.de first, the only account you need. Credentials are used for the download and not stored; --skip-gated opts out. The benchmark datasets live separately under $GRAFT_BENCHMARK_ROOT, see docs/EVALUATION.md.

---

Checkpoints

Both load into the same GRAFTModel and differ only in configuration.

drop_vis_feats pushes every image sample point out of bounds, so the refinement sees scene geometry only — the mode the paper uses as a plug-in prior on top of other methods, refining an existing reconstruction with no retraining and no image features. The two also use different flat_hand_mean conventions, so from_pretrained needs an explicit GRAFTConfig for a bare checkpoint path.

---

The scene image

GRAFT takes two views of the same camera: the photo with the people, and a view of the same scene without them, which is where the geometry comes from. Both benchmarks supply a real one. For your own images, best first:

model.predict("photo.jpg", scene_image="empty_room.jpg")   # a real empty-scene photo

from graft.scene import remove_people                      # or inpaint them out
model.predict("photo.jpg", scene_image=remove_people("photo.jpg"))

model.predict("photo.jpg")                                 # fallback: no scene view

Inpainting needs the demo extra, plus Hub access to the gated FLUX.1-dev and Omnieraser (huggingface-cli login). remove_people sizes FLUX's 31 GiB to the free VRAM automatically — it fits a 24 GiB card, slowly; see offload= for the strategies. Passing no scene view at all reuses the photo as its own, so the people occlude the geometry being fitted: a fallback, not a supported setting.

---

Evaluation

export GRAFT_BENCHMARK_ROOT=/path/to/datasets    # holds PROX/
python evaluate.py

PROX and RICH are distributed by their own authors and not redistributed here; docs/EVALUATION.md says what to download and how to lay it out. Six metrics per benchmark — PA-MPJPE (mm), vertex-to-scene L2 (mm) and angle (degrees), contact precision / recall / F1:

The graft rows are the published numbers, reproduced exactly. The geometry-only rows differ from the paper's ablation by ~2 mm PA-MPJPE, which predates the batched body fit used here; the interaction metrics are essentially unchanged. Scored frames are fewer than the totals (178 and 100) where no single-person prediction is made.

---

Performance

python benchmark.py --image photo.jpg --scene-image empty_room.jpg

Inference latency is 174 ms (5.7 images/s) on one H100, for a single image with one person and three refinement iterations:

Measured from prepared input tensors, so image decoding is excluded. benchmark.py exits non-zero if the median exceeds --budget (200 ms default).

---

Licence

Our code is MIT (see LICENSE). The released weights are for non-commercial research use only — GRAFT loads MapAnything (CC BY-NC 4.0), NLF and SMPL-X at run time, all of which are non-commercial.

Vendored third-party code keeps its own licence: third_party/mapanything/ is Apache-2.0 (Meta), with our changes recorded in third_party/PATCHES.md.

Citation

If you find our work useful, please consider citing:

@misc{ym2026graft,
  title        = {GRAFT: Geometric Refinement and Fitting Transformer for Human Scene Reconstruction},
  author       = {Pradyumna YM and Yuxuan Xue and Yue Chen and Nikita Kister and Istv{\'a}n S{\'a}r{\'a}ndi and Gerard Pons-Moll},
  year         = {2026},
  eprint       = {2604.19624},
  archivePrefix = {arXiv},
  url          = {https://arxiv.org/abs/2604.19624},
}

Acknowledgements

GRAFT builds directly on MapAnything (scene geometry), NLF (initialisation), MoGe (metric scale), smplfitter (body fitting), and optionally WiLoR (hands) and OmniEraser (scene inpainting). See third_party/PATCHES.md.