Repository Overview

This repository houses the code implementation for the technical report of the 3D Reconstruction From Monocular Multi-Food Images Challenge at the CVPR 2025 MetaFood Workshop.

Get Started

SAM Installation

pip install -r ./SAM/requirements.txt

Hunyuan3Dv2 Installation

pip install -r ./Hunyuan3D-2/requirements.txt

Usage

Data Preparation

.
└── food_mask
    ├── 1-bagel_cream_cheese
    ├── 2-breaded_fish_lemon_broccoli
    ├── 3-burger_hot_dog
    ├── 4-cheesecake_strawberry_raspberry
    ├── 5-energy_bar_cheddar_cheese_banana
    ├── 6-grilled_salmon_broccoli
    ├── 7-pasta_garlic_bread
    ├── 8-pbj_carrot_stick_apple_celery
    ├── 9-pizza_chicken_wing
    ├── 10-quesadilla_guacamole_salsa
    ├── 11-roast_chicken_leg_biscuit
    ├── 12-sandwich_cookie
    ├── 13-steak_mashed_potatoes
    └── 14-toast_sausage_fried_egg

Plate (Reference Object) Segmentation

python ./SAM/plate_sam.py

Food Segmentation (Per Food Item)

Modify the api_key in food_sam.py before running:

python ./SAM/food_sam.py

Output Data Format After Execution:

.
└── food_mask
    ├── 1-bagel_cream_cheese
    │   ├── 1.jpeg
    │   ├── 1_boxes_filt.txt    # Bounding box for the plate
    │   ├── bagel.jpg
    │   ├── bagel.txt           # Bounding box for bagel
    │   ├── bagel_cropped.jpg   # Segmented bagel image
    │   ├── cream_cheese.jpg
    │   ├── cream_cheese.txt    # Bounding box for cream cheese
    │   ├── cream_cheese_cropped.jpg  # Segmented cream cheese image
    │   └── plate.jpg
    ├── 2-breaded_fish_lemon_broccoli
    ├── 3-burger_hot_dog
    ├── 4-cheesecake_strawberry_raspberry
    ├── 5-energy_bar_cheddar_cheese_banana
    ├── 6-grilled_salmon_broccoli
    ├── 7-pasta_garlic_bread
    ├── 8-pbj_carrot_stick_apple_celery
    ├── 9-pizza_chicken_wing
    ├── 10-quesadilla_guacamole_salsa
    ├── 11-roast_chicken_leg_biscuit
    ├── 12-sandwich_cookie
    ├── 13-steak_mashed_potatoes
    └── 14-toast_sausage_fried_egg

3D Reconstruction

python mask_big.py
python ./Hunyuan3D-2/food.py

Scale Estimation

python scale_estimation.py

• Adjust real_plate for the actual width of the reference plate (e.g., a physical plate’s width).
• Tune the perspective scaling factor r as needed.

Volume and Chamfer Distance Metrics

Segmentation Metric Calculation

We evaluate food segmentation performance (beyond competition metrics like volume and Chamfer Distance) using manually annotated bounding boxes as ground truth (GT). GT data structure:

.
└── food_mask_GT
    ├── 1-bagel_cream_cheese
    ├── 2-breaded_fish_lemon_broccoli
    ├── 3-burger_hot_dog
    ├── 4-cheesecake_strawberry_raspberry
    ├── 5-energy_bar_cheddar_cheese_banana
    ├── 6-grilled_salmon_broccoli
    ├── 7-pasta_garlic_bread
    ├── 8-pbj_carrot_stick_apple_celery
    ├── 9-pizza_chicken_wing
    ├── 10-quesadilla_guacamole_salsa
    ├── 11-roast_chicken_leg_biscuit
    ├── 12-sandwich_cookie
    ├── 13-steak_mashed_potatoes
    └── 14-toast_sausage_fried_egg

Calculate IOU for segmentation:

python IOU.py

Segmentation IOU Results

🤗Acknowledgement

This work is built on many amazing research works and open-source projects, thanks a lot to all the authors for sharing!

• Grounded-Segment-Anything
• Hunyuan3D-2
• Qwen-VL