ϕ-RIE

From Photorealistic Reconstruction to Interactive Environments

Project page · Paper · Citation · Demo video · PhiView code

ϕ-RIE turns captured rooms into editable environments. It connects object discovery, 3D asset generation, metric registration, background completion, and physics. PhiView lets you explore the result, change object parameters, shoot projectiles, and run robot interactions.

Runyi Yang, Deheng Zhang, Xiaoye Wang, Kanzhi Wu, Lei Sun, Ajad Chhatkuli, Kunyu Peng*, Luc Van Gool, Danda Paudel

INSAIT, Sofia University “St. Kliment Ohridski” · vivo Mobile Communication Co., Ltd. · Karlsruhe Institute of Technology

Contact: [email protected]

*Corresponding author

Image demos

Watch the full demo or explore all nine recordings on the project page.

Method overview

The pipeline connects captured Gaussian scenes to complete object assets and backgrounds, then keeps rendered appearance aligned with simulated body poses.

1. Installation

Start with the control tools

Install uv, then:

git clone https://github.com/insait-institute/PhiRIE.git
cd PhiRIE
uv sync --project envs/control --locked
bash run/phiroom.sh modules

The control environment lists components, prepares commands, and runs CPU tools. The checkout includes the PhiView source in tools/phiview/.

Set up reconstruction and simulation

GPU reconstruction and rendering require an NVIDIA GPU and the runtime for the selected model. Prepare the environments, data, and weights with these guides:

2. Prepare data, weights and environments

Datasets live under data/, run products under outputs/, vendored checkouts under third_party/, and downloaded weights under weights/ and checkpoints/; all of them are gitignored. Every location is resolved by run/env.sh and agents/core/common.py and can be overridden with an environment variable.

Python environments

The GPU stages need separate interpreters because their torch builds are mutually incompatible. run/env.sh resolves them and exposes one helper per environment (run <package>.<module> [args]):

1. Create .venv/ (Python 3.11) and run bash run/setup_env.sh. It installs torch 2.4.1+cu124, open3d, trimesh, coacd, pybullet, transformers, xformers, spconv, gsplat, Depth Anything 3 and simple-lama-inpainting, clones third_party/TRELLIS, and ends with an import smoke test.
2. Create the sam3 environment (torch >= 2.5, diffusers >= 0.39, the sam3 package) and the gsplat environment (Python 3.10, prebuilt gsplat CUDA wheel); point SIMANY_SAM3_PY and SIMANY_GSPLAT_PY at them.
3. The CPU control environment for run/phiroom.sh is locked: uv sync --project envs/control --locked. The phiroom CLI reads the same variables, or explicit paths from configs/runtime.local.json (copy configs/runtime.example.json).

run/env.sh sets HF_HUB_OFFLINE=1 and TORCH_CUDA_ARCH_LIST="8.6;9.0+PTX" unless they are exported first, and honours HF_HOME and TORCH_HOME. Environment setup explains why the environments cannot be merged and lists the optional ones.

Datasets

data/ is an empty mount point. Select a scene with SIMANY_SCENE; outputs go to SIMANY_OUT (default outputs/<scene> plus a mode suffix). Scenes reconstructed from a phone video by run/run_video2sim.sh are written in the same layout under data/recon_scenes/, so no dataset is needed for them.

Model weights

Hugging Face models load offline at run time; prefetch them once with HF_HUB_OFFLINE=0 (HF_HOME relocates the cache):

Other weights: the LaMa fallback big-lama.pt goes to ~/.cache/torch/hub/checkpoints/ (run/run_inpaint.sh exports LAMA_MODEL); DreamSim (a ReconViaGen dependency) downloads about 1.9 GB into weights/; VGGT-Omega weights go to checkpoints/vggt-omega/ (gated, see agents/models/vggt_scene.py); SAM 3D Objects reads third_party/sam-3d-objects/checkpoints/hf/pipeline.yaml (SIMANY_SAM3D_CONFIG). For pi0.5, set OPENPI_ROOT to a checkout of xuningy/openpi with its uv environment; OPENPI_DATA_HOME (default checkpoints/openpi_cache/) holds openpi-assets-simeval/<name>, SIMANY_PI05_CKPT selects the checkpoint subpath and SIMANY_PI05_CONFIG=pi05_droid_jointpos_sim the sim-co-trained variant. Run bash run/pi05_serve.sh on a GPU machine with generous host RAM.

Third-party checkouts

third_party/ is gitignored: TRELLIS/ (cloned by run/setup_env.sh; SIMANY_TRELLIS_DIR), mujoco_menagerie/ (git clone https://github.com/google-deepmind/mujoco_menagerie third_party/mujoco_menagerie; SIMANY_MUJOCO_MENAGERIE_ROOT), ReconViaGen/ (SIMANY_RVG_DIR), MaskClustering/ and FlashSplat/ (baselines), sam-3d-objects/ and vggt-omega/ (optional generators and pose backends), and BEHAVIOR-1K/ (OmniGibson export; robo/sim/omnigibson_bridge/install_omnigibson.sh). See third_party/README.md.

Check the installation

bash run/smoke_imports.sh
bash run/phiroom.sh modules
bash run/phiroom.sh plan generation trellis -- --help

3. Basic demos

Watch or download the recordings

The recordings are stored in media/; the demo gallery on the project page plays the same files in the browser.

Try the browser playground

The project page hosts scene editing, a physics sandbox, and a recorded robot replay. It runs in a WebGPU-capable browser; the recordings above remain available everywhere.

Inspect a component before running it

bash run/phiroom.sh describe generation
bash run/phiroom.sh plan generation trellis -- --help
bash run/phiroom.sh describe phiview

plan shows the command and required runtime. Replace plan with run when the model environment, input data, and weights are ready.

Open a prepared scene in PhiView

After completing PhiView setup, initialize your configuration and launch a prepared scene:

bash run/phiview.sh init --config configs/phiview.local.yaml \
  --simany-root "$PWD/tools/phiview/backends/simany" \
  --outputs-root "$PWD/outputs" \
  --data-root /path/to/scannetpp --splats-root /path/to/splats

bash run/phiview.sh run viewer --action demo \
  --config configs/phiview.local.yaml --scene ROOM_factory

Select an object, make it simulatable, then use the viewer's shooting, friction, and movement controls. The harmonizer can be enabled separately.

4. Components

Scene reconstruction

Build the scene representation from captured views, camera geometry, meshes, and 3D Gaussian splats. See reconstruction.

Object discovery and generation

Find objects in the captured scene and generate complete asset candidates with the available model adapters. See discovery and generation.

Metric registration

Align each candidate with its observed object so position, orientation, and scale agree with the scene. See registration.

Background completion

Remove the selected object's original appearance and fill its exposed background before moving the replacement. See inpainting.

Physics and PhiView

Connect collision geometry, mass, friction, and Gaussian appearance. The example below compares masses of 0.15 and 0.75 kg with friction coefficients of 0.05 and 0.80. See physics and PhiView.

Robot interaction

Render the robot alongside reconstructed objects and use the simulation interfaces for interaction. The recordings above show scripted grasp-and-place sequences. Policy integration and evaluation tools are documented in robotics.

Appearance harmonization

Apply optional harmonization to the rendered view while preserving the recorded camera path and physical state. See the harmonizer integration and matched-view comparison.

5. Build a scene

Prepare posed images, the scene mesh, and a Gaussian reconstruction using the data guide. Configure the paths in your environment, then run the automatic construction pipeline:

export SIMANY_SCENE=YOUR_SCENE_ID
export SIMANY_OUT="$PWD/outputs/${SIMANY_SCENE}_auto"
bash run/run_auto.sh

This runs object discovery, crop preparation, asset generation, registration, collision construction, and simulator export. For background editing, run:

bash run/run_inpaint.sh

See the pipeline guide for inputs, stage outputs, and the separate benchmark and single-image workflows.

6. Code map

The Python package and command remain named phiroom for compatibility with earlier versions. Development uses main.

7. Release plan

Releases are tagged on main; changes are listed in CHANGELOG.md.

License

PhiRIE is released under the MIT License. PhiView (tools/phiview/) is a separate MIT-licensed project bundled here with its license and provenance, and the model checkouts under third_party/ keep their own upstream licenses.

Citation

Paper on arXiv

@misc{yang2026phirie,
  title = {{$\phi$-RIE}: From Photorealistic Reconstruction to Interactive Environments},
  author = {Runyi Yang and Deheng Zhang and Xiaoye Wang and Kanzhi Wu and Lei Sun and Ajad Chhatkuli and Kunyu Peng and Luc Van Gool and Danda Pani Paudel},
  year = {2026},
  eprint = {2609.26795},
  archivePrefix = {arXiv},
  primaryClass = {cs.RO},
  url = {https://arxiv.org/abs/2609.26795}
}