MVHOI

MVHOI is a lightweight research-code release for multi-view hand-object interaction reconstruction. It combines multi-view object reconstruction, multi-view hand regression, joint 3D optimization, and anti-penetration refinement:

stacked_npy 6-view HOI
  -> HSV hand/object segmentation
  -> SD LoRA dual-task inpainting
  -> InstantMesh multi-view object reconstruction
  -> OmniHands multi-view MANO regression
  -> mesh repair + SDF
  -> multi-view HOI optimization
  -> Stage 4 selective anti-penetration refinement

The repository is intentionally lightweight for GitHub. Datasets, model checkpoints, pretrained third-party weights, and generated meshes are not committed; see data/README.md, weights/README.md, docs/MODEL_ZOO.md, docs/WEIGHT_PACKAGE_MANIFEST.md, and the reproducibility checklist in docs/RUN_BASELINE.md.

Demo

Input 6-view rendered HOI sample:

Multi-view hand silhouette alignment used by the optimization:

Stage-wise hand-object optimization result:

What This Release Contains

• src/: MVHOI optimization code plus multi-view/OmniHands extensions.
• preprocess/: segmentation, inpainting, rendering, InstantMesh, and training/inference utilities.
• tools/: metric computation and table composition scripts.
• assets/: small MANO/contact helper assets; MANO model files must be downloaded separately.
• figures/metrics/: published quantitative metric JSON/Markdown tables, including SV vs MV ablation.
• docs/: pipeline, data, model, training, evaluation, and third-party notes.

Environment

The original project uses several environments because dependencies conflict:

• mvhoi: optimization, mesh repair, OmniHands fine-tuning/evaluation.
• instantmesh: InstantMesh inference/fine-tuning.
• lisa: LISA segmentation for real images.
• sd_inpaint or equivalent diffusers env: SD inpainting LoRA training/inference.

Main environment:

conda create -n mvhoi python=3.9
conda activate mvhoi
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda env update --file environment.yaml

Install PyTorch3D, MANO, InstantMesh, OmniHands, HaMeR/ViTPose, and optional LISA following docs/THIRD_PARTY.md.

Required Paths

Set these environment variables before running training or full evaluation:

export MVHOI_CONTACT_ROOT=/path/to/contact_processed
export MVHOI_HO3D_ROOT=/path/to/HO3D_v2/train
export MVHOI_WEIGHTS_ROOT=$PWD/weights
export MVHOI_MANO_ROOT=$MVHOI_WEIGHTS_ROOT/mano_assets
export MVHOI_THIRD_PARTY_ROOT=$PWD/third_party
export MVHOI_OMNI_ROOT=$MVHOI_THIRD_PARTY_ROOT/OmniHands
export MVHOI_INSTANTMESH_ROOT=$MVHOI_THIRD_PARTY_ROOT/InstantMesh
export MVHOI_HAMER_DATA_ROOT=$MVHOI_WEIGHTS_ROOT/hamer_data
export MVHOI_OMNI_BACKBONE=$MVHOI_WEIGHTS_ROOT/omnihands/Demo_Multiview.pth
export MVHOI_OMNI_CONFIG=$MVHOI_WEIGHTS_ROOT/omnihands/config_multi.yaml
export PYOPENGL_PLATFORM=egl

Expected data and weight layouts are documented in docs/DATA_LAYOUT.md and docs/MODEL_ZOO.md. For the script-by-script baseline run, use docs/RUN_BASELINE.md.

Run Multi-View Optimization

python src/optim_mvhoi.py \
  --seq ABF11 --frame 0070 --view 0 \
  --npy_path data/stacked_npy/ABF11/normalized_0070.npy \
  --data_our data/examples/ABF11_0070 \
  --obj_mesh data/examples/ABF11_0070/obj_recon/results/instantmesh/instant-mesh-large/meshes/ABF11_0070_sdlora/fixed.obj \
  --out_dir outputs/mv_optim/ABF11_0070 \
  --omni_ckpt weights/omnihands/epoch58-mpvpe1.63.ckpt \
  --omni_mode mv \
  --supervision_views 0,1,2,3,4,5

The default runtime path uses SD-completed hand silhouettes for Stage 1-3 supervision, R_norm=identity, automatic hand scale search, and the Stage 2 depth guard. Use --r_norm_source transform or --stage_mask_source omnihands_render only for diagnostics or ablations.

For the SV baseline:

python tools/run_sv_pipeline.py
python tools/compute_phase2_metrics.py --pred_root outputs/sv_mvo_optim --out_suffix _sv
python tools/compute_group3_metrics.py --pred_root outputs/sv_mvo_optim --out_suffix _sv
python tools/compute_hand_quality_per_stage.py --pred_root outputs/sv_mvo_optim --out_suffix _sv
python tools/compose_sv_vs_mv_table.py

Current Quantitative Result

The 9-sample HO3D SV vs MV ablation is in figures/metrics/sv_vs_mv_table.md.

Attribution

Cite InstantMesh, OmniHands, HaMeR, ViTPose, LISA, MOHO, and IHOI when using their components.