ForeAct: Steering Your VLA with Efficient Visual Foresight Planning

Paper | Model | Real-World Data

News

• [2026/04] 🔥 We make a coffee making demo.
• [2026/04] 🔥 We support multi-view visual foresight generation with per-view customizable resolution.
• [2026/02] We release the code and model for ForeAct!

About

• ForeAct is a visual foresight planner that empowers VLAs with the ability to anticipate future observations, enabling more informed decision-making.
• ForeAct is general and plug-and-play: state-of-the-art VLAs can seamlessly incorporate ForeAct without any architectural modification.
• ForeAct is highly efficient, generating a high-fidelity 640 $\times$ 480 future observation in just 0.33s on a single H100 GPU.

Demo

Usage

Environment Setup

git clone https://github.com/mit-han-lab/foreact
cd foreact
bash environment_setup.sh foreact

Finetune

Download the pretrained weights and prepare your own real-world data (or use our processed real-world data). Update the relevant paths in the config you want to run (configs/finetune.yaml for single-view, configs/finetune_multiview.yaml for multi-view), then launch:

### Single-view
bash scripts/run_finetune.sh finetune.yaml

### Multi-view
bash scripts/run_finetune.sh finetune_multiview.yaml

Inference

### CLI (single-view)
python app_cli.py --checkpoint_path path/to/model --prompt "" \
    --input_image primary=path/to/image --output_dir ./results

### CLI (multi-view)
python app_cli.py --checkpoint_path path/to/model --prompt "" \
    --input_image primary=path/to/img1 wrist_left=path/to/img2 wrist_right=path/to/img3 \
    --output_dir ./results

### Gradio (auto-adapts to the checkpoint's view configuration)
python app.py --checkpoint_path path/to/model

VLA Training

We provide examples regarding policy training in ./third-party/lerobot.

Acknowledgements

Thanks to metaquery, diffusers, lerobot for the wonderful open-source codebase.