Affostruction

Official code and data for:

Affostruction: 3D Affordance Grounding with Generative Reconstruction
Chunghyun Park¹, Seunghyeon Lee¹, and Minsu Cho^1,2
1POSTECH and ²RLWRLD
CVPR 2026, Denver.

"Reconstruct what's hidden, ground where to interact."

Updates

• 2026-09-01 — released the training and evaluation code: data preparation, both training stages, and the three benchmark protocols.
• 2026-05-13 — initial release: inference code, pretrained checkpoints and demos.

Installation

Requires Linux x86_64 + CUDA 12.4.

curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync

Prefix commands with uv run; no venv activation needed.

Pretrained checkpoints

Auto-downloaded from chrockey/Affostruction on first run; SLAT + mesh/gaussian decoders come from microsoft/TRELLIS-image-large. The tables below compare the released checkpoints against the numbers reported in the paper, using the protocols in Evaluation. Flow-matching sampling is stochastic, so small deviations between runs are expected.

3D reconstruction

Toys4k, 1,250 test objects, single RGB-D view.

Complete 3D affordance grounding

Affogato, 1,000 test objects, affordance flow on ground-truth geometry.

Partial 3D affordance grounding

Affogato, 1,000 test objects, affordance flow on the reconstruction from a single RGB-D view.

Usage

from affostruction import AffostructionPipeline

pipeline = AffostructionPipeline.from_pretrained().cuda()
outputs = pipeline.run(input_dict, queries=["Point to the part you would sit on."])

coords = outputs["coords"]                  # (N, 4) sparse voxel coords
probs  = outputs["affordance"][0]["probs"]  # (N,) per-voxel heatmap in [0, 1], paired with coords

uv run python examples/affostruction.py --data_dir examples/data/sample2

Need mesh / gaussian? Pass formats=["mesh", "gaussian"] — decoding is opt-in.

Reconstruction only? Drop queries. See examples/reconstruction.py for more details.

Data preparation

Training and evaluation run on TRELLIS-format datasets: Affogato for affordance supervision, plus 3D-FUTURE / ABO / HSSD for reconstruction and Toys4k for the reconstruction benchmark. dataset_toolkits/ ships a patch against the upstream TRELLIS toolkit (depth-enabled conditioning renders) together with the Affogato-specific scripts.

# upstream toolkit + our delta
git clone https://github.com/microsoft/TRELLIS.git /tmp/TRELLIS
cp -rn /tmp/TRELLIS/dataset_toolkits/* dataset_toolkits/
git apply --directory=dataset_toolkits dataset_toolkits/trellis.patch

ln -s /path/to/your/datasets trellis_data   # processed datasets live here

Then follow DATASET.md, which gives the exact per-dataset commands and the resulting directory layout.

Training

Prepare data first — see DATASET.md. Both stages train on 8 GPUs (batch 8/GPU), AdamW lr 1e-4, EMA 0.9999, 10% condition dropout for CFG. WandB logging is optional (set WANDB_API_KEY; project via WANDB_PROJECT).

# Stage 1 — multi-view RGBD sparse-structure flow (validates on Toys4k)
torchrun --nproc_per_node=8 --standalone train.py \
    --config configs/stage1_reconstruction.json \
    --output_dir outputs/stage1_reconstruction \
    --data_dir trellis_data/3d-future,trellis_data/abo,trellis_data/hssd,trellis_data/affogato

# Stage 2 — text-conditioned affordance heatmap flow (Affogato only)
torchrun --nproc_per_node=8 --standalone train.py \
    --config configs/stage2_affordance.json \
    --output_dir outputs/stage2_affordance \
    --data_dir trellis_data/affogato

Checkpoints land in ckpts/denoiser_ema*.pt every i_save steps — set save_best_only to keep only the best-validation snapshot — and training resumes automatically from misc_step*.pt. A training output dir can be evaluated directly (below).

Evaluation

Test manifests (object lists, sha256↔UID mappings) live in manifests/. --ckpt takes either an HF repo id or a local training output dir.

1. Predictions. predict.py runs the reconstruction pipeline over a dataset test split (1 view = frame 0, seed 1, TRELLIS sampler defaults) and writes outputs.npz (sparse structure) per object — plus mesh.glb with --save_mesh, needed for Chamfer / F-score.

python predict.py --ckpt chrockey/Affostruction --datasets toys4k --num_views 1 --save_mesh \
    --gt_renders_dir results/benchmark/toys4k/gt/recon_renders
python predict.py --ckpt chrockey/Affostruction --datasets affogato   --num_views 1

2. Metrics.

# 3D reconstruction (Toys4k, 1,250 objects): volumetric IoU at 64^3
python eval.py --task recon --data_dir trellis_data/toys4k \
    --pred_dir results/benchmark/toys4k/Affostruction/1

# + Chamfer distance, F-score@0.05, PSNR and LPIPS (see DATASET.md §3 for the
# ground-truth point clouds and renders)
python eval.py --task recon --data_dir trellis_data/toys4k \
    --pred_dir results/benchmark/toys4k/Affostruction/1 \
    --mesh_metrics --gt_points_dir results/benchmark/toys4k/gt/points \
    --appearance_metrics --gt_renders_dir results/benchmark/toys4k/gt/recon_renders

# complete-input affordance grounding: aIoU / AUC / SIM / MAE on the annotation
# point clouds, affordance flow conditioned on ground-truth geometry
python eval.py --task complete --ckpt chrockey/Affostruction \
    --data_dir trellis_data/affogato --annotations_dir /path/to/affogato_raw/annotations

# partial-input affordance grounding: aIoU / aCD on the *reconstructed* voxels
python eval.py --task partial --ckpt chrockey/Affostruction \
    --data_dir trellis_data/affogato --pred_dir results/benchmark/affogato/Affostruction/1

Affordance sampling defaults to 50 steps, cfg 1.0, noise scale 0.1 and a zero negative condition; --all_queries switches the partial protocol from the first query per object to all five. --seed (default 0) is recorded in the result JSON.

Acknowledgments

Built on TRELLIS by Microsoft.

Citation

@inproceedings{park2026affostruction,
  title={Affostruction: 3D Affordance Grounding with Generative Reconstruction},
  author={Park, Chunghyun and Lee, Seunghyeon and Cho, Minsu},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2026}
}