PAT3D: Physics-Augmented Text-to-3D Scene Generation

Overview

PAT3D generates physically grounded simulation-ready 3D scenes from text prompts. The release code is organized around the dashboard pipeline: a web UI launches a staged Python worker, the worker calls provider modules for each stage, and outputs are written as structured runtime JSON plus mesh, rendered image, and metric.

Dashboard Website Overview

The dashboard website provides one place to configure a text-to-3D scene run, launch the staged pipeline, monitor live progress, inspect intermediate artifacts, view the final scene, and compute metrics.

Pipeline

The dashboard runs the paper-core pipeline stage by stage:

1. Reference image generation: create the anchor image from the scene prompt.
2. Scene understanding: estimate depth and segment requested objects.
3. Object relation extraction: describe objects, infer support/containment relations, and estimate size priors.
4. Object asset generation: create or load per-object 3D assets.
5. Layout initialization: place objects into an initial semantic scene layout.
6. Simulation preparation: convert layout assets into low-poly simulation-ready meshes.
7. Physics simulation: run forward simulation or diff-sim initialization plus forward simulation.
8. Visualization: export scene bundles and rendered previews for the dashboard viewer.
9. Metrics: render metric views and compute semantic scores for generated scenes.

Backend Routing

Most dashboard stages use a single release backend. The useful routing summary is:

Requirements

PAT3D is maintained around two supported install paths:

Install these host tools before setting up PAT3D:

• Git
• Node.js 20 or newer and npm
• NVIDIA driver with CUDA 13 support
• Docker Engine plus NVIDIA Container Toolkit if you are using the Docker path
• Blender 4.x for rendered previews and scene export

Additional components used by the full PAT3D workflow:

DepthPro expects the pretrained checkpoint at extern/ml-depth-pro/checkpoints/depth_pro.pt by default. You can override it with PAT3D_DEPTH_PRO_CHECKPOINT if the checkpoint lives elsewhere.

The bundled pyuipc-0.9.0-cp310-cp310-linux_x86_64.whl is built for the supported Ubuntu 24.04 target. On older glibc targets such as Ubuntu 22.04, the native launch path may fail to load the private physics backend. In that case, use the Docker path or provide a compatible replacement wheel through PAT3D_PRIVATE_PHYSICS_WHEEL.

Installation

Run all commands from the repository root.

The canonical installers are scripts/setup_install.py for native installs and scripts/setup_docker_install.sh for Docker installs. Both produce .env.install, which is consumed by the launch scripts.

1. Clone the repository

git clone https://github.com/Simulation-Intelligence/PAT3D
cd PAT3D

If your release uses Git submodules, initialize them:

git submodule update --init --recursive

If the external repositories are distributed separately, place them under extern/ with these expected paths:

Install these external repositories before running the full dashboard workflow. extern/t2v_metrics is only required when you want VQAScore metrics.

2. Configure credentials and local paths

Create .env:

cp .env.example .env

Set these values before running either installer:

HF_TOKEN=<token with access to facebook/sam3 and facebook/sam3.1>

PAT3D includes a prebuilt Diff_GIPC / libuipc wheel at private_wheels/pyuipc-0.9.0-cp310-cp310-linux_x86_64.whl, and both setup scripts use that tracked wheel automatically.

You also need an OpenAI key before launching the dashboard:

OPENAI_API_KEY=<your_key>

Common optional overrides:

OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_MODEL=gpt-5.4
OPENAI_IMAGE_MODEL=gpt-image-1.5
BLENDER_BIN=/path/to/blender
PAT3D_HF_TOKEN_FILE=/absolute/path/to/hf_token.txt
PAT3D_PRIVATE_PHYSICS_WHEEL=/absolute/path/to/alternate/pyuipc-0.9.0-cp310-cp310-linux_x86_64.whl

3. Choose an install path

Native install

Use this when the host already matches the supported Linux/NVIDIA target:

python3 scripts/setup_install.py

The native installer provisions:

• .venv for the main PAT3D runtime
• .venv-sam3 for SAM 3
• extern/Hunyuan3Dv2 if the checkout is missing
• extern/fTetWild if the checkout is missing
• extern/sam3
• checkpoints/depth_pro.pt
• checkpoints/sam3/config.json and checkpoints/sam3/sam3.pt
• extern/fTetWild/build-local/FloatTetwild_bin
• Hunyuan native extensions (custom_rasterizer, custom_rasterizer_kernel, and mesh_processor)
• dashboard npm dependencies and a Playwright browser
• .env.install with the derived PAT3D path variables

Launch the dashboard:

bash scripts/run_dashboard.sh

Docker install

Use this when you want the supported PAT3D runtime to live inside the bundled container image:

bash scripts/setup_docker_install.sh

Then launch the dashboard from the installed Docker environment:

bash scripts/run_docker_dashboard.sh

The Docker path:

• build docker/Dockerfile.install if needed
• run the installer inside the supported CUDA 13 / Ubuntu 24.04 image with --gpus all
• mount the repo, the private physics wheel, and persistent local caches
• run scripts/setup_install.py --skip-system-deps inside the container
• reuse the installed repo-local .venv, .venv-sam3, checkpoints, and caches when serving the dashboard

If you only want to build the reusable image first:

bash scripts/build_docker_image.sh

4. Optional integrations

• Install Blender 4.x for rendered scene previews and exports.
• For VQAScore metrics, add extern/t2v_metrics and set:

PAT3D_T2V_METRICS_ROOT=/absolute/path/to/extern/t2v_metrics
PAT3D_VQA_PYTHON=/absolute/path/to/vqa/python
PAT3D_T2V_HF_CACHE_DIR=/absolute/path/to/data/metrics/hf_cache

• The generated .env.install file already records PAT3D_DASHBOARD_PYTHON, PAT3D_SAM3_PYTHON, PAT3D_SAM3_CHECKPOINT_PATH, and PAT3D_FTETWILD_BIN. You usually do not need to set them manually.
• The dashboard writes generated scene data under results/workspaces/<scene_id>/..., with runtime metadata, job state, and metrics under results/runtime/, results/dashboard_jobs/, and results/metrics/.

Run the Dashboard

If you used the native install:

bash scripts/run_dashboard.sh

If you used the Docker install:

bash scripts/run_docker_dashboard.sh

For manual development wiring only, you can also start the dashboard server with:

npm --prefix dashboard run dev

Open:

http://127.0.0.1:4173

Typical dashboard flow:

1. Enter a scene prompt, for example A small wooden chair next to a round side table in a simple room.
2. Optionally enter a scene name, for example demo_chair_table.
3. Review the requested objects and choose automatic or manual segmentation.
4. Configure physics settings. By default, diff-sim is off and the pipeline uses forward simulation settings.
5. Click Generate.
6. Watch the stage progress panel and open the log panel if a stage fails.
7. Inspect the final scene in the browser viewer.
8. Run metrics for the selected output when you need semantic evaluation.

Metrics

Semantic Metrics

Physical Metrics

Citation

@inproceedings{
      lin2026patd,
      title={{PAT}3D: Physics-Augmented Text-to-3D Scene Generation},
      author={Guying Lin and Kemeng Huang and Michael Liu and Ruihan Gao and Hanke Chen and Lyuhao Chen and Beijia Lu and Taku Komura and Yuan Liu and Jun-Yan Zhu and Minchen Li},
      booktitle={The Fourteenth International Conference on Learning Representations},
      year={2026},
      url={https://openreview.net/forum?id=iIRxFkeCuY}
      }

Acknowledgement

The simulation part of this project is developed based on the open-source project libuipc. We sincerely thank all the authors of that project.