Installation

1. Clone the repo and create conda environment

git clone https://github.com/ZhenglinZhou/DreamDPO.git
cd DreamDPO
conda create -n dreamdpo python=3.9
conda activate dreamdpo
pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu118
pip install xformers==0.0.23.post1
pip install ninja
git clone https://github.com/bytedance/MVDream extern/MVDream
pip install -e extern/MVDream 
pip install -r requirements.txt

2. Install the pretrained reward model

• Step1: Install the packages as following:

• Step2: Download model checkpoints:

pip install -U "huggingface_hub[cli]"
huggingface-cli login
huggingface-cli download ${Model CheckPoints} --local-dir model_weights/${Reward Model}

• Step3: Organize them look like this:

|-- model_weights
    |-- HPSv2 # HPSv2
        |-- HPS_v2.1_compressed.pt
    |-- ImageReward # ImageReward
        |-- ImageReward.pt
        |-- med_config.json
    |-- Reward3D # Reward3D
        |-- Reward3D_Scorer.pt
    |-- CLIP-ViT-H-14-laion2B-s32B-b79K # HPSv2
        |-- open_clip_pytorch_model.bin

• Step4 (Optional): Use large multi-module model (e.g., Qwen): To use it, you should create an Alibaba Cloud account and create a Dashscope API key to fill in the qwen_api_key field in the threestudio/reward/qwen.py file.

Usage

1. DreamDPO with different reward models, including hpsv2-score, imagereward-score, brique-score, reward3d-score:

python3 launch.py --config configs/dreamdpo/mvdream-sd21-reward.yaml \
--train --gpu 0 \
system.prompt_processor.prompt="A pair of hiking boots caked with mud at the doorstep of a cabin" \
system.guidance.beta_dpo=0.01 system.guidance.reward_model="hpsv2-score"

2. DreamDPO with large multi-module model (e.g., Qwen)

python3 launch.py --config configs/dreamdpo/mvdream-sd21-lmm.yaml \
--train --gpu 0 \
system.prompt_processor.prompt="A pair of hiking boots caked with mud at the doorstep of a cabin" \
system.guidance.ai_start_iter=1200 system.guidance.ai_prob=0.5

Acknowledgments

• This code is built on the threestudio-project, MVDream, and AppAgent. Thanks to the maintainers!
• We also borrow code and model from HPSv2, ImageReward, DreamReward. Thanks to the authors for their human preference alignment models.

Citation

If you find DreamDPO useful for your research and applications, please cite us using this BibTeX:

@inproceedings{zhou2025dreamdpo,
      title={DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization}, 
      author={Zhenglin Zhou and Xiaobo Xia and Fan Ma and Hehe Fan and Yi Yang and Tat-Seng Chua},
      booktitle={ICML},
      year={2025},
}