VLM_3D_Scene

Reproduce a reference scene photo as a single Blender scene from a set of component images. The project has two stages:

• Stage A — Component generation. Turn each image in Assets/Images/ into a 3D .glb with Pixal3D (image-to-3D).
• Stage B — Layout pipeline. A step-based language-agent pipeline (agents/) that places the generated GLBs into one scene matching the reference photo, then exports assembled_scene.blend.

Repository layout

Prerequisites

• Python 3.10+, git, git-lfs, and Blender on PATH (Stage B calls blender --background --python).
• An OpenAI-compatible API (set model.base_url in configs/scene_layout.yaml to point at your endpoint, e.g. a local proxy at http://127.0.0.1:8199/v1).
• CUDA GPU for Stage A (Pixal3D).

Stage A — generate components

pip install -r Pixal3D/requirements.txt        # Pixal3D deps
./scripts/run_generate_components.sh            # GLBs -> outputs/components/

This generates one .glb per image in Assets/Images/ (config in configs/generation.yaml).

Stage B — run the layout pipeline

pip install openai>=1.0 pyyaml
export OPENAI_API_KEY=sk-...

python agents/pipeline.py --config configs/scene_layout.yaml

The pipeline runs 7 sequential steps (camera calibration → correspondence → scales → scene graph → incremental layout → global refinement → Blender assembly). Each step reads its spec from tasks/0N_*.html, iterates with a render-and-compare loop, and writes canonical JSON outputs consumed by later steps. See agents/README.md for the full step table, CLI flags, the iteration-status contract, and run records.

Useful flags:

python agents/pipeline.py --start-step 5        # resume from a step
python agents/pipeline.py --only-step 6         # run a single step
python agents/pipeline.py --only-step 7 --regenerate-assembler
python agents/pipeline.py --continue-on-failure

Notes

• Generated heavy artifacts (outputs/, .blend, caches) are git-ignored; the source images and code are committed.
• The Stage B transform convention is pure Blender-native: location in meters, rotation_euler in radians (XYZ Euler), scale unitless.