Awesome-Physics-aware-Generation

😄😄 Under Construction 😄😄

🚀 Physics-Aware Generation

🚀 Physics Engine/Simulation Platforms

🚀 Physics Simulation

🚀 Physics Understanding (from Videos/Observations)

🚀 Physics Evaluation

---

Physics-Aware Generation

1️Image Modal Generation🖼️

1. Physically Compatible 3D Object Modeling from a Single Image. Minghao Guo, Bohan Wang, Pingchuan Ma, Tianyuan Zhang, Crystal Elaine Owens, Chuang Gan, Joshua B. Tenenbaum, Kaiming He, Wojciech Matusik, arXiv, 2024. Paper

2. Generative Photography: Scene-Consistent Camera Control for Realistic Text-to-Image Synthesis. Yuan Yu, Xijun Wang, Yichen Sheng, Prateek Chennuri, Xingguang Zhang, and Stanley Chan, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. GitHub

3. Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control. Hassan Abu Alhaija, Jose Alvarez, Maciej Bala, Tiffany Cai, Tianshi Cao, Liz Cha, Joshua Chen, Mike Chen, Francesco Ferroni, Sanja Fidler, Dieter Fox, Yunhao Ge, Jinwei Gu, Ali Hassani, Michael Isaev, Pooya Jannaty, Shiyi Lan, Tobias Lasser, Huan Ling, Ming-Yu Liu, Xian Liu, Yifan Lu, Alice Luo, Qianli Ma, Hanzi Mao, Fabio Ramos, Xuanchi Ren, Tianchang Shen, Shitao Tang, Ting-Chun Wang, Jay Wu, Jiashu Xu, Stella Xu, Kevin Xie, Yuchong Ye, Xiaodong Yang, Xiaohui Zeng, Yu Zeng, arXiv, 2025. GitHub

4. Feature Splatting: Language-Driven Physics-Based Scene Synthesis and Editing. Ri-Zhao Qiu, Ge Yang, Weijia Zeng, Xiaolong Wang, arXiv, 2024. Paper

5. Disco4D: Disentangled 4D Human Generation and Animation from a Single Image. Hui En Pang, Shuai Liu, Zhongang Cai, Lei Yang, Tianwei Zhang, Ziwei Liu, arXiv, 2024. Paper

6. Generative Image Dynamics. Zhengqi Li, Richard Tucker, Noah Snavely, Aleksander Holynski, arXiv, 2023. Paper

7. LivePhoto: Real Image Animation with Text-guided Motion Control. Xi Chen, Zhiheng Liu, Mengting Chen, Yutong Feng, Yu Liu, Yujun Shen, Hengshuang Zhao, arXiv, 2023. Paper

8. Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators. Daniel Geng, Andrew Owens, International Conference on Learning Representations (ICLR), 2024. GitHub

9. Implicit Warping for Animation with Image Sets. Arun Mallya, Ting-Chun Wang, Ming-Yu Liu, Advances in Neural Information Processing Systems (NeurIPS), 2022. Paper

10. Thin-Plate Spline Motion Model for Image Animation. Jian Zhao, Hui Zhang, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. Paper

11. Controllable Animation of Fluid Elements in Still Images. Aniruddha Mahapatra, Kuldeep Kulkarni, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. Paper

12. Animating Pictures with Eulerian Motion Fields. Aleksander Holynski, Brian Curless, Steven M Seitz, Richard Szeliski, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. Paper

13. MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model. Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, Mike Zheng Shou, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. Homepage

14. Phy124: Fast Physics-Driven 4D Content Generation from a Single Image. Jiajing Lin, Zhenzhong Wang, Yongjie Hou, Yuzhou Tang, Min Jiang, arXiv, 2024. Paper

15. PhyCAGE: Physically Plausible Compositional 3D Asset Generation from a Single Image. Han Yan, Mingrui Zhang, Yang Li, Chao Ma, Pan Ji, arXiv, 2024. Paper

16. PhysMotion: Physics-Grounded Dynamics From a Single Image. Xiyang Tan, Ying Jiang, Xuan Li, Zeshun Zong, Tianyi Xie, Yin Yang, Chenfanfu Jiang, arXiv, 2024. Homepage

17. PID: Physics-Informed Diffusion Model for Infrared Image Generation. Fangyuan Mao, Jilin Mei, Shun Lu, Fuyang Liu, Liang Chen, Fangzhou Zhao, Yu Hu, arXiv, 2024. GitHub

18. SINGAPO: Single Image Controlled Generation of Articulated Parts in Objects. Jiayi Liu, Denys Iliash, Angel X. Chang, Manolis Savva, Ali Mahdavi-Amiri, arXiv, 2024. Paper

19. Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering. Kim Youwang, Tae-Hyun Oh, Gerard Pons-Moll, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. GitHub

20. Learning an Implicit Physics Model for Image-based Fluid Simulation. Emily Yue-Ting Jia, Jiageng Mao, Zhiyuan Gao, Yajie Zhao, and Yue Wang, In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025. Paper

2️⃣Video Modal Generation🎥

1. Unsupervised Learning for Physical Interaction through Video Prediction. Chelsea Finn, Ian Goodfellow Openai, Sergey Levine, Google Brain, Advances in Neural Information Processing Systems (NeurIPS), 2016. Paper

2. Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control. Hassan Abu Alhaija, Jose Alvarez, Maciej Bala, Tiffany Cai, Tianshi Cao, Liz Cha, Joshua Chen, Mike Chen, Francesco Ferroni, Sanja Fidler, Dieter Fox, Yunhao Ge, Jinwei Gu, Ali Hassani, Michael Isaev, Pooya Jannaty, Shiyi Lan, Tobias Lasser, Huan Ling, Ming-Yu Liu, Xian Liu, Yifan Lu, Alice Luo, Qianli Ma, Hanzi Mao, Fabio Ramos, Xuanchi Ren, Tianchang Shen, Shitao Tang, Ting-Chun Wang, Jay Wu, Jiashu Xu, Stella Xu, Kevin Xie, Yuchong Ye, Xiaodong Yang, Xiaohui Zeng, Yu Zeng, arXiv, 2025. GitHub

3. Compositional 3D-aware Video Generation with LLM Director. Hanxin Zhu, Tianyu He, Anni Tang, Junliang Guo, Zhibo Chen, Jiang Bian, arXiv, 2024. Paper

4. VideoComposer: Compositional Video Synthesis with Motion Controllability. Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, Jingren Zhou, arXiv, 2023. Paper

5. SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction. Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, Ziwei Liu, arXiv, 2023. GitHub

6. InterDyn: Controllable Interactive Dynamics with Video Diffusion Models. Rick Akkerman, Haiwen Feng, Michael J. Black, Dimitrios Tzionas, Victoria Fernández Abrevaya, arXiv, 2024. Paper

7. VideoPoet: A Large Language Model for Zero-Shot Video Generation. Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, Krishna Somandepalli, Hassan Akbari, Yair Alon, Yong Cheng, Josh Dillon, Agrim Gupta, Meera Hahn, Anja Hauth, David Hendon, Alonso Martinez, David Minnen, Mikhail Sirotenko, Kihyuk Sohn, Xuan Yang, Hartwig Adam, Ming-Hsuan Yang, Irfan Essa, Huisheng Wang, David A. Ross, Bryan Seybold, Lu Jiang, International Conference on Machine Learning (ICML), 2024. Paper

8. Understanding Object Dynamics for Interactive Image-to-Video Synthesis. Andreas Blattmann, Timo Milbich, Michael Dorkenwald, Björn Ommer, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. Paper

9. AutoVFX: Physically Realistic Video Editing from Natural Language Instructions. Hao-Yu Hsu, Zhi-Hao Lin, Albert Zhai, Hongchi Xia, Shenlong Wang, arXiv, 2024. Homepage

10. Controllable Video Generation Through Global and Local Motion Dynamics. Aram Davtyan, Paolo Favaro, European Conference on Computer Vision, 2022. Paper

11. Generating 3D-Consistent Videos from Unposed Internet Photos. Gene Chou, Kai Zhang, Sai Bi, Hao Tan, Zexiang Xu, Fujun Luan, Bharath Hariharan, Noah Snavely, arXiv, 2024. Paper

12. Generative Omnimatte: Learning to Decompose Video into Layers. Yao-Chih Lee, Erika Lu, Sarah Rumbley, Michal Geyer, Jia-Bin Huang, Tali Dekel, Forrester Cole, arXiv, 2024. Homepage

13. GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planning. Jiaxi Lv, Yi Huang, Mingfu Yan, Jiancheng Huang, Jianzhuang Liu, Yifan Liu, Yafei Wen, Xiaoxin Chen, Shifeng Chen, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. GitHub

14. HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness. Zihui Xue, Mi Luo, Changan Chen, Kristen Grauman, arXiv, 2024. GitHub

15. Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback. Hiroki Furuta, Heiga Zen, Dale Schuurmans, Aleksandra Faust, Yutaka Matsuo, Percy Liang, Sherry Yang, arXiv, 2024. Paper

16. Layered Controllable Video Generation. Jiahui Huang, Yuhe Jin, Kwang Moo Yi, Leonid SIgal, European Conference on Computer Vision (ECCV), 2022. Homepage

17. Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generation. Aram Davtyan, Paolo Favaro, Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2024. GitHub

18. Motion Prompting: Controlling Video Generation with Motion Trajectories. Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Tatiana Lopez-Guevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, Chen Sun, Oliver Wang, Andrew Owens, Deqing Sun, arXiv, 2024. Homepage

19. Motion-Conditioned Diffusion Model for Controllable Video Synthesis. Tsai-Shien Chen, Chieh Hubert Lin, Hung-Yu Tseng, Tsung-Yi Lin, Ming-Hsuan Yang, arXiv, 2023. Paper

20. MotionCraft: Physics-based Zero-Shot Video Generation. Luca Savant Aira, Antonio Montanaro, Emanuele Aiello, Diego Valsesia, Enrico Magli, arXiv, 2024. GitHub

21. PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Prediction. Hao Wu, Wei Xiong, Fan Xu, Xiao Luo, Chong Chen, Xian-Sheng Hua, Haixin Wang, Proceedings of the 32nd ACM International Conference on Multimedia, 2024. Paper

22. PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation. Tianyuan Zhang, Hong-Xing Yu, Rundi Wu, Brandon Y. Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, William T. Freeman, European Conference on Computer Vision (ECCV), 2024. GitHub

23. PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation. Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, Shenlong Wang, European Conference on Computer Vision (ECCV), 2024. GitHub

24. Physics-based Human Motion Estimation and Synthesis from Videos. Kevin Xie, Tingwu Wang, Umar Iqbal, Yunrong Guo, Sanja Fidler, Florian Shkurti, Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. Paper

25. Scene Co-pilot: Procedural Text to Video Generation with Human in the Loop. Zhaofang Qian, Abolfazl Sharifi, Tucker Carroll, Ser-Nam Lim, arXiv, 2024. Homepage

26. Motion Dreamer: Realizing Physically Coherent Video Generation through Scene-Aware Motion Reasoning. Tianshuo Xu, Zhifei Chen, Leyi Wu, Hao Lu, Yuying Chen, Lihui Jiang, Bingbing Liu, Yingcong Chen, arXiv, 2024. Paper

27. Towards Physically Stable Motion Generation: A New Paradigm of Human Pose Representation. Qiongjie Cui, Zhenyu Lou, Zhenbo Song, Xiangbo Shu, IEEE Transactions on Circuits and Systems for Video Technology, 2024. Paper

28. Generating Physically Realistic and Directable Human Motions from Multi-Modal Inputs. Aayam Shrestha, Pan Liu, German Ros, Kai Yuan, Alan Fern, European Conference on Computer Vision (ECCV), 2024. Homepage

29. Learning Plug-and-play Memory for Guiding Video Diffusion Models.. Selena Song, Ziming Xu, Zijun Zhang, Kun Zhou, Jiaxian Guo, Lianhui Qin, and Biwei Huang, arXiv, 2025. Homepage

30. VDAWorld: World Modelling via VLM-Directed Abstraction and Simulation.. Felix O'Mahony, Roberto Cipolla, and Ayush Tewari, arXiv, 2025. Homepage

31. VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models.. Xiangdong Zhang, Jiaqi Liao, Shaofeng Zhang, Fanqing Meng, Xiangpeng Wan, Junchi Yan, and Yu Cheng, arXiv, 2025. Homepage

32. VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Prior.. Xindi Yang, Baolu Li, Yiming Zhang, Zhenfei Yin, Lei Bai, Liqian Ma, Zhiyong Wang et al, arXiv, 2025. Homepage

33. Synthetic Video Enhances Physical Fidelity in Video Synthesis.. Qi Zhao, Xingyu Ni, Ziyu Wang, Feng Cheng, Ziyan Yang, Lu Jiang, and Bohan Wang, arXiv, 2025. Homepage

34. Planning with Sketch-Guided Verification for Physics-Aware Video Generation.. Huang, Yidong, Zun Wang, Han Lin, Dong-Ki Kim, Shayegan Omidshafiei, Jaehong Yoon, Yue Zhang, and Mohit Bansal, arXiv, 2025. Homepage

35. ProPhy: Progressive Physical Alignment for Dynamic World Simulation.. Zijun Wang, Panwen Hu, Jing Wang, Terry Jingchen Zhang, Yuhao Cheng, Long Chen, Yiqiang Yan, Zutao Jiang, Hanhui Li, and Xiaodan Liang, arXiv, 2025. Paper

36. Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach.. Yunuo Chen, Junli Cao, Vidit Goel, Sergei Korolev, Chenfanfu Jiang, Jian Ren, Sergey Tulyakov, and Anil Kag, arXiv, 2025. Homepage

37. Hierarchical Fine-Grained Preference Optimization for Physically Plausible Video Generation.. Harold Haodong Chen, Haojian Huang, Qifeng Chen, Harry Yang, and Ser-Nam Lim, arXiv, 2025. Homepage

38. PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation.. Chen Wang, Chuhao Chen, Yiming Huang, Zhiyang Dou, Yuan Liu, Jiatao Gu, and Lingjie Liu, arXiv, 2025. Homepage

39. PhysChoreo: Physics-Controllable Video Generation with Part-Aware Semantic Grounding.. Haoze Zhang, Tianyu Huang, Zichen Wan, Xiaowei Jin, Hongzhi Zhang, Hui Li, and Wangmeng Zuo, arXiv, 2025. Paper

40. PhysMaster: Mastering Physical Representation for Video Generation via Reinforcement Learning.. Sihui Ji, Xi Chen, Xin Tao, Pengfei Wan, and Hengshuang Zhao, arXiv, 2025. Homepage

41. What about Gravity in Video Generation? Post-Training Newton's Laws with Verifiable Rewards.. Minh-Quan Le, Yuanzhi Zhu, Vicky Kalogeiton, and Dimitris Samaras, arXiv, 2025. Homepage

42. NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamics.. Yu Yuan, Xijun Wang, Tharindu Wickremasinghe, Zeeshan Nadir, Bole Ma, and Stanley H. Chan, arXiv, 2025. Homepage

43. MoReGen: Multi-Agent Motion-Reasoning Engine for Code-based Text-to-Video Synthesis.. Xiangyu Bai, He Liang, Bishoy Galoaa, Utsav Nandi, Shayda Moezzi, Yuhang He, and Sarah Ostadabbas, arXiv, 2025. Paper

44. LINA: Learning INterventions Adaptively for Physical Alignment and Generalization in Diffusion Models.. Shu Yu, and Chaochao Lu, arXiv, 2025. Homepage

45. Think Before You Diffuse: LLMs-Guided Physics-Aware Video Generation.. Ke Zhang, Cihan Xiao, Yiqun Mei, Jiacong Xu, and Vishal M. Patel, arXiv, 2025. Homepage

46. Reasoning Physical Video Generation with Diffusion Timestep Tokens via Reinforcement Learning.. Wang Lin, Liyu Jia, Wentao Hu, Kaihang Pan, Zhongqi Yue, Wei Zhao, Jingyuan Chen, Fei Wu, and Hanwang Zhang, arXiv, 2025. Paper

47. Bootstrapping Physics-Grounded Video Generation through VLM-Guided Iterative Self-Refinement.. Yang Liu, Xilin Zhao, Peisong Wen, Siran Dai, and Qingming Huang, arXiv, 2025. Paper

3️⃣3D Modal Generation🧊

1. Visual Grounding of Learned Physical Models. Yunzhu Li, Toru Lin, Kexin Yi, Daniel M. Bear, Daniel L. K. Yamins, Jiajun Wu, Joshua B. Tenenbaum, Antonio Torralba, International Conference on Machine Learning (ICML), 2020. GitHub

2. GASP: Gaussian Splatting for Physic-Based Simulations. Piotr Borycki, Weronika Smolak, Joanna Waczyńska, Marcin Mazur, Sławomir Tadeja, Przemysław Spurek, arXiv, 2024. Paper

3. Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control. Hassan Abu Alhaija, Jose Alvarez, Maciej Bala, Tiffany Cai, Tianshi Cao, Liz Cha, Joshua Chen, Mike Chen, Francesco Ferroni, Sanja Fidler, Dieter Fox, Yunhao Ge, Jinwei Gu, Ali Hassani, Michael Isaev, Pooya Jannaty, Shiyi Lan, Tobias Lasser, Huan Ling, Ming-Yu Liu, Xian Liu, Yifan Lu, Alice Luo, Qianli Ma, Hanzi Mao, Fabio Ramos, Xuanchi Ren, Tianchang Shen, Shitao Tang, Ting-Chun Wang, Jay Wu, Jiashu Xu, Stella Xu, Kevin Xie, Yuchong Ye, Xiaodong Yang, Xiaohui Zeng, Yu Zeng, arXiv, 2025. GitHub

4. VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality. Ying Jiang, Chang Yu, Tianyi Xie, Xuan Li, Yutao Feng, Huamin Wang, Minchen Li, Henry Lau, Feng Gao, Yin Yang, Chenfanfu Jiang, Proceedings - SIGGRAPH 2024 Conference Papers, 2024. Paper

5. DoughNet: A Visual Predictive Model for Topological Manipulation of Deformable Objects. Dominik Bauer, Zhenjia Xu, Shuran Song, European Conference on Computer Vision (ECCV), 2024. GitHub

6. Improving Physics-Augmented Continuum Neural Radiance Field-Based Geometry-Agnostic System Identification with Lagrangian Particle Optimization. Takuhiro Kaneko, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. Paper

7. PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification. Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chenfanfu Jiang, Chuang Gan, International Conference on Learning Representations (ICLR), 2023. Paper

8. 3D-VLA: A 3D Vision-Language-Action Generative World Model. Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, Chuang Gan, arXiv, 2024. GitHub

9. AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers. Sherwin Bahmani, Ivan Skorokhodov, Guocheng Qian, Aliaksandr Siarohin, Willi Menapace, Andrea Tagliasacchi, David B. Lindell, Sergey Tulyakov, arXiv, 2024. Homepage

10. Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabrication. Yunuo Chen, Tianyi Xie, Zeshun Zong, Xuan Li, Feng Gao, Yin Yang, Ying Nian Wu, Chenfanfu Jiang, arXiv, 2024. Paper

11. Automated 3D Physical Simulation of Open-world Scene with Gaussian Splatting. Haoyu Zhao, Hao Wang, Xingyue Zhao, Hongqiu Wang, Zhiyu Wu, Chengjiang Long, Hua Zou, arXiv, 2024. Paper

12. DiffuseBot: Breeding Soft Robots With Physics-Augmented Generative Diffusion Models. Tsun-Hsuan Wang, Juntian Zheng, Pingchuan Ma, Yilun Du, Byungchul Kim, Andrew Spielberg, Joshua Tenenbaum, Chuang Gan, Daniela Rus, Advances in Neural Information Processing Systems (NeurIPS), 2023. Homepage

13. DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors. Tianyu Huang, Haoze Zhang, Yihan Zeng, Zhilu Zhang, Hui Li, Wangmeng Zuo, Rynson W. H. Lau, arXiv, 2024. GitHub

14. LIVE-GS: LLM Powers Interactive VR by Enhancing Gaussian Splatting. Haotian Mao, Zhuoxiong Xu, Siyue Wei, Yule Quan, Nianchen Deng, Xubo Yang, arXiv, 2024. Paper

15. PhyRecon: Physically Plausible Neural Scene Reconstruction. Junfeng Ni, Yixin Chen, Bohan Jing, Nan Jiang, Bin Wang, Bo Dai, Puhao Li, Yixin Zhu, Song-Chun Zhu, Siyuan Huang, arXiv, 2024. GitHub

16. Physics informed neural fields for smoke reconstruction with sparse data. Mengyu Chu, Lingjie Liu, Quan Zheng, Erik Franz, Hans Peter Seidel, Christian Theobalt, Rhaleb Zayer, ACM Transactions on Graphics, 2022. Paper

17. Physics-Guided Human Motion Capture with Pose Probability Modeling. Jingyi Ju, Buzhen Huang, Chen Zhu, Zhihao Li, Yangang Wang, arXiv, 2023. GitHub

18. Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis. Yuanhao Cai, Yixun Liang, Jiahao Wang, Angtian Wang, Yulun Zhang, Xiaokang Yang, Zongwei Zhou, Alan Yuille, arXiv, 2024. GitHub

19. RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation. Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, Chuang Gan, arXiv, 2023. GitHub

20. Synthetic Vision: Training Vision-Language Models to Understand Physics. Vahid Balazadeh, Mohammadmehdi Ataei, Hyunmin Cheong, Amir Hosein Khasahmadi, Rahul G. Krishnan, arXiv, 2024. Paper

21. VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control. Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, David B. Lindell, Sergey Tulyakov, arXiv, 2024. Homepage

22. VividDream: Generating 3D Scene with Ambient Dynamics. Yao-Chih Lee, Yi-Ting Chen, Andrew Wang, Ting-Hsuan Liao, Brandon Y. Feng, Jia-Bin Huang, arXiv, 2024. Paper

23. Physically-aware Generative Network for 3D Shape Modeling. Mariem Mezghanni, Malika Boulkenafed, André Lieutier, Maks Ovsjanikov, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. Paper

24. Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos. Kun Su, Kaizhi Qian, Eli Shlizerman, Antonio Torralba, Chuang Gan, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. Homepage

25. Procedural Generation of Videos to Train Deep Action Recognition Networks. Cesar Roberto deSouza, Adrien Gaidon, Yohann Cabon, Antonio Manuel Lopez Pena, arXiv, 2016. Paper

26. SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models. Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, Bo Dai, arXiv, 2023. Paper

27. StableV2V: Stablizing Shape Consistency in Video-to-Video Editing. Chang Liu, Rui Li, Kaidong Zhang, Yunwei Lan, Dong Liu, arXiv, 2024. Paper

28. StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaboration. Panwen Hu, Jin Jiang, Jianqi Chen, Mingfei Han, Shengcai Liao, Xiaojun Chang, Xiaodan Liang, arXiv, 2024. Paper

29. Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation. Fanqing Meng, Jiaqi Liao, Xinyu Tan, Wenqi Shao, Quanfeng Lu, Kaipeng Zhang, Yu Cheng, Dianqi Li, Yu Qiao, Ping Luo, arXiv, 2024. GitHub

30. Trajectory Optimization for Physics-Based Reconstruction of 3d Human Pose from Monocular Video. Erik Gartner, Mykhaylo Andriluka, Hongyi Xu, Cristian Sminchisescu, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. Homepage

31. Video Creation by Demonstration. Yihong Sun, Hao Zhou, Liangzhe Yuan, Jennifer J. Sun, Yandong Li, Xuhui Jia, Hartwig Adam, Bharath Hariharan, Long Zhao, Ting Liu, arXiv, 2024. GitHub

32. Video2Game: Real-time, Interactive, Realistic and Browser-Compatible Environment from a Single Video. Hongchi Xia, Zhi-Hao Lin, Wei-Chiu Ma, Shenlong Wang, arXiv, 2024. Homepage

33. Physically-guided Disentangled Implicit Rendering for 3D Face Modeling. Zhenyu Zhang, Yanhao Ge, Ying Tai, Weijian Cao, Renwang Chen, Kunlin Liu, Hao Tang, Xiaoming Huang, Chengjie Wang, Zhifeng Xie, Dongjin Huang, Tencent Youtu Lab, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. Paper

34. Few-Shot Physically-Aware Articulated Mesh Generation via Hierarchical Deformation. Xueyi Liu, Bin Wang, He Wang, Yi Li, Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. GitHub

35. LLplace: The 3D Indoor Scene Layout Generation and Editing via Large Language Model. Yixuan Yang, Junru Lu, Zixiang Zhao, Zhen Luo, James J. Q. Yu, Victor Sanchez, Feng Zheng, arXiv, 2024. Paper

36. PhysPart: Physically Plausible Part Completion for Interactable Objects. Rundong Luo, Haoran Geng, Congyue Deng, Puhao Li, Zan Wang, Baoxiong Jia, Leonidas Guibas, Siyuan Huang, arXiv, 2024. Paper

37. PHYSCENE: Physically Interactable 3D Scene Synthesis for Embodied AI. Yandan Yang, Baoxiong Jia, Peiyuan Zhi, Siyuan Huang, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. Paper

38. SOPHY: Generating Simulation-Ready Objects with Physical Materials.. Junyi Cao, and Evangelos Kalogerakis, arXiv, 2025. Homepage

4️⃣4D Modal Generation⏳

1. Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imagination. Leonardo Barcellona, Andrii Zadaianchuk, Davide Allegro, Samuele Papa, Stefano Ghidoni, Efstratios Gavves, arXiv, 2024. Paper

2. Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control. Hassan Abu Alhaija, Jose Alvarez, Maciej Bala, Tiffany Cai, Tianshi Cao, Liz Cha, Joshua Chen, Mike Chen, Francesco Ferroni, Sanja Fidler, Dieter Fox, Yunhao Ge, Jinwei Gu, Ali Hassani, Michael Isaev, Pooya Jannaty, Shiyi Lan, Tobias Lasser, Huan Ling, Ming-Yu Liu, Xian Liu, Yifan Lu, Alice Luo, Qianli Ma, Hanzi Mao, Fabio Ramos, Xuanchi Ren, Tianchang Shen, Shitao Tang, Ting-Chun Wang, Jay Wu, Jiashu Xu, Stella Xu, Kevin Xie, Yuchong Ye, Xiaodong Yang, Xiaohui Zeng, Yu Zeng, arXiv, 2025. GitHub

3. Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models. Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, Karsten Kreis, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. Paper

4. LoopGaussian: Creating 3D Cinemagraph with Multi-view Images via Eulerian Motion Field. Jiyang Li, Lechao Cheng, Zhangye Wang, Tingting Mu, Jingxuan He, arXiv, 2024. Paper

5. PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics. Tianyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Li, Yutao Feng, Yin Yang, Chenfanfu Jiang, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. GitHub

6. GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation. Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, Hanbo Zhang, Minzhao Zhu, arXiv, 2024. Homepage

7. Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptions. Zhenyu Jiang, Yuqi Xie, Jinhan Li, Ye Yuan, Yifeng Zhu, Yuke Zhu, arXiv, 2024. Paper

8. LLMPhy: Complex Physical Reasoning Using Large Language Models and World Models. Anoop Cherian, Radu Corcodel, Siddarth Jain, Diego Romeres, arXiv, 2024. Paper

9. Phys4DGen: A Physics-Driven Framework for Controllable and Efficient 4D Content Generation from a Single Image. Jiajing Lin, Zhenzhong Wang, Shu Jiang, Yongjie Hou, Min Jiang, arXiv, 2024. Homepage

10. PhysDiff: Physics-Guided Human Motion Diffusion Model. Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, Jan Kautz, Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. Paper

11. Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusion. Fangfu Liu, Hanyang Wang, Shunyu Yao, Shengjun Zhang, Jie Zhou, Yueqi Duan, arXiv, 2024. Homepage

12. Scaling In-the-Wild Training for Diffusion-based Illumination Harmonization and Editing by Imposing Consistent Light Transport. Unknown, International Conference on Learning Representations (ICLR), 2025. GitHub

13. M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenes. Sixu Yan, Zeyu Zhang, Muzhi Han, Zaijin Wang, Qi Xie, Zhitian Li, Zhehan Li, Hangxin Liu, Xinggang Wang, Song-Chun Zhu, arXiv, 2024. Paper

14. PhysReaction: Physically Plausible Real-Time Humanoid Reaction Synthesis via Forward Dynamics Guided 4D Imitation. Yunze Liu, Changxi Chen, Chenjing Ding, Li Yi, arXiv, 2024. Paper

15. MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators. Shenghai Yuan*, Jinfa Huang*, Yujun Shi, Yongqi Xu, Ruijie Zhu, Bin Lin, Xinhua Cheng, Li Yuan, Jiebo Luo, arXiv, 2024. Homepage Paper Code

16. Articulated Kinematics Distillation from Video Diffusion Models.. Xuan Li, Qianli Ma, Tsung-Yi Lin, Yongxin Chen, Chenfanfu Jiang, Ming-Yu Liu, and Donglai Xiang, In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), 2025. Paper

Physics Engine/ Simulation Platforms

1. Genesis: A Generative and Universal Physics Engine for Robotics and Beyond. Genesis Authors, arXiv, 2024. Homepage

2. Pymunk. Pymunk Authors, arXiv, 2024. Website

3. Taichi: A language for high-performance computation on spatially sparse data structures. Yuanming Hu, Tzu Mao Li, Luke Anderson, Jonathan Ragan-Kelley, Frédo Durand, ACM Transactions on Graphics, 2019. Paper

4. DiffTaichi: Differentiable Programming for Physical Simulation. Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, Frédo Durand, arXiv, 2019. Paper

5. MuJoCo: A physics engine for model-based control. Emanuel Todorov, Tom Erez, Yuval Tassa, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012. Paper

6. FluidLab: A Differentiable Environment for Benchmarking Complex Fluid Manipulation. Zhou Xian, Bo Zhu, Zhenjia Xu, Hsiao-Yu Tung, Antonio Torralba, Katerina Fragkiadaki, Chuang Gan, arXiv, 2023. Paper

7. SAPIEN: A SimulAted Part-based Interactive ENvironment. Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, Hao Su, Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2020. Paper

8. ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation. Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Michael Lingelbach, Aidan Curtis, Kevin Feigelis, Daniel M. Bear, Dan Gutfreund, David Cox, Antonio Torralba, James J. DiCarlo, Joshua B. Tenenbaum, Josh H. McDermott, Daniel L. K. Yamins, arXiv, 2020. Paper

9. UBSoft: A Simulation Platform for Robotic Skill Learning in Unbounded Soft Environments. Chunru Lin, Jugang Fan, Yian Wang, Zeyuan Yang, Zhehuan Chen, Lixing Fang, Tsun-Hsuan Wang, Zhou Xian, Chuang Gan, arXiv, 2024. Paper

10. Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning. Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, Gavriel State, arXiv, 2021. Paper

11. PR2: A Physics- and Photo-realistic Testbed for Embodied AI and Humanoid Robots. Hangxin Liu, Qi Xie, Zeyu Zhang, Tao Yuan, Xiaokun Leng, Lining Sun, Song-Chun Zhu, Jingwen Zhang, Zhicheng He, Yao Su, arXiv, 2024. Paper

12. PyBullet. PyBullet Authors, arXiv, 2024. Website

13. Nvidia PhysX. Nvidia PhysX Authors, arXiv, 2024. GitHub

14. Open Dynamics Engine. Russ Smith, arXiv, 2024. Website

15. Chrono: An open source multi-physics dynamics engine. Alessandro Tasora, Radu Serban, Hammad Mazhar, Arman Pazouki, Daniel Melanz, Jonathan Fleischmann, Michael Taylor, Hiroyuki Sugiyama, Dan Negrut, High Performance Computing in Science and Engineering, 2015. Paper

16. Unity: A General Platform for Intelligent Agents. Arthur Juliani, Vincent-Pierre Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, Danny Lange, arXiv, 2018. Paper

17. Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation. C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, Olivier Bachem, arXiv, 2021. Paper

18. Design and use paradigms for gazebo, an open-source multi-robot simulator. N. Koenig, A. Howard, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566), 2004. Paper

19. WebotsTM: Professional Mobile Robot Simulation. Olivier Michel, arXiv, 2004. Paper

20. XPBD: Position-based simulation of compliant constrained dynamics. Miles Macklin, Matthias Müller, Nuttapong Chentanez, Proceedings - Motion in Games 2016: 9th International Conference on Motion in Games, MIG 2016, 2016. Paper

Physics Simulation

1. Genesis: A Generative and Universal Physics Engine for Robotics and Beyond. Genesis Authors, arXiv, 2024. Homepage

2. Pymunk. Pymunk Authors, arXiv, 2024. Website

3. Taichi: A language for high-performance computation on spatially sparse data structures. Yuanming Hu, Tzu Mao Li, Luke Anderson, Jonathan Ragan-Kelley, Frédo Durand, ACM Transactions on Graphics, 2019. Paper

4. DiffTaichi: Differentiable Programming for Physical Simulation. Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, Frédo Durand, arXiv, 2019. Paper

5. ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation. Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Michael Lingelbach, Aidan Curtis, Kevin Feigelis, Daniel M. Bear, Dan Gutfreund, David Cox, Antonio Torralba, James J. DiCarlo, Joshua B. Tenenbaum, Josh H. McDermott, Daniel L. K. Yamins, arXiv, 2020. Paper

6. UBSoft: A Simulation Platform for Robotic Skill Learning in Unbounded Soft Environments. Chunru Lin, Jugang Fan, Yian Wang, Zeyuan Yang, Zhehuan Chen, Lixing Fang, Tsun-Hsuan Wang, Zhou Xian, Chuang Gan, arXiv, 2024. Paper

7. Efficient Generation of Multimodal Fluid Simulation Data. Daniele Baieri, Donato Crisostomi, Stefano Esposito, Filippo Maggioli, Emanuele Rodolà, arXiv, 2023.Paper

8. Learning to Simulate Complex Physics with Graph Networks. Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, Peter W. Battaglia, International Conference on Machine Learning (ICML), 2020. Paper

9. Complex Locomotion Skill Learning via Differentiable Physics. Yu Fang, Jiancheng Liu, Mingrui Zhang, Jiasheng Zhang, Yidong Ma, Minchen Li, Yuanming Hu, Chenfanfu Jiang, Tiantian Liu, arXiv, 2022. Paper

10. Differentiable Simulation of Soft Multi-body Systems. Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, Ming C. Lin, Advances in Neural Information Processing Systems (NeurIPS), 2021. GitHub

11. DiffPD: Differentiable Projective Dynamics. Tao Du, Kui Wu, Pingchuan Ma, Sebastien Wah, Andrew Spielberg, Daniela Rus, Wojciech Matusik, ACM Transactions on Graphics, 2022. Paper

12. PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics. Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B. Tenenbaum, Chuang Gan, International Conference on Learning Representations (ICLR), 2021. Paper

13. Graph networks as learnable physics engines for inference and control. Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, Peter Battaglia, International Conference on Machine Learning (ICML), 2018. Paper

14. Differentiable Simulation of Soft Multi-body Systems. Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, Ming C. Lin, Advances in Neural Information Processing Systems (NeurIPS), 2021. Paper

15. DIFFTACTILE: A Physics-based Differentiable Tactile Simulator for Contact-rich Robotic Manipulation. Zilin Si, Gu Zhang, Qingwei Ben, Branden Romero, Zhou Xian, Chao Liu, Chuang Gan, arXiv, 2024. Paper

16. Efficient Differentiable Simulation of Articulated Bodies. Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, Ming C. Lin, International Conference on Machine Learning (ICML), 2021. Paper

17. Interpretable Intuitive Physics Model. Tian Ye, Xiaolong Wang, James Davidson, Abhinav Gupta, Proceedings of the European Conference on Computer Vision (ECCV), 2018. Paper

18. Learning to Identify Physical Parameters from Video Using Differentiable Physics. Rama Krishna Kandukuri, Jan Achterhold, Michael Möller, Jörg Stückler, arXiv, 2020. Paper

19. Scalable Differentiable Physics for Learning and Control. Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, Ming C. Lin, arXiv, 2020. Paper

20. InfiniteWorld: A Unified Scalable Simulation Framework for General Visual-Language Robot Interaction. Pengzhen Ren, Min Li, Zhen Luo, Xinshuai Song, Ziwei Chen, Weijia Liufu, Yixuan Yang, Hao Zheng, Rongtao Xu, Zitong Huang, Tongsheng Ding, Luyang Xie, Kaidong Zhang, Changfei Fu, Yang Liu, Liang Lin, Feng Zheng, Xiaodan Liang, arXiv, 2024. Paper

21. DiffXPBD : Differentiable Position-Based Simulation of Compliant Constraint Dynamics. Tuur Stuyck, Hsiao-yu Chen, arXiv, 2023. Paper

22. Unified simulation of elastic rods, shells, and solids. Sebastian Martin, Peter Kaufmann, Mario Botsch, Eitan Grinspun, Markus Gross, ACM SIGGRAPH 2010 Papers, SIGGRAPH 2010, 2010. Paper

Physics Understanding (from Videos/Observations)

1. Learning to Simulate Complex Physics with Graph Networks. Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, Peter W. Battaglia, International Conference on Machine Learning (ICML), 2020. Paper

2. Graph networks as learnable physics engines for inference and control. Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, Peter Battaglia, International Conference on Machine Learning (ICML), 2018. Paper

3. Interpretable Intuitive Physics Model. Tian Ye, Xiaolong Wang, James Davidson, Abhinav Gupta, Proceedings of the European Conference on Computer Vision (ECCV), 2018. Paper

4. Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussians. Licheng Zhong, Hong-Xing Yu, Jiajun Wu, Yunzhu Li, European Conference on Computer Vision (ECCV), 2024. GitHub

5. GIC: Gaussian-Informed Continuum for Physical Property Identification and Simulation. Junhao Cai, Yuji Yang, Weihao Yuan, Yisheng He, Zilong Dong, Liefeng Bo, Hui Cheng, Qifeng Chen, Advances in Neural Information Processing Systems (NeurIPS), 2024. GitHub

6. Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video. Boxiang Rong, Artur Grigorev, Wenbo Wang, Michael J. Black, Bernhard Thomaszewski, Christina Tsalicoglou, Otmar Hilliges, arXiv, 2024. Paper

7. PIE-NeRF: Physics-based Interactive Elastodynamics with NeRF. Yutao Feng, Yintong Shang, Xuan Li, Tianjia Shao, Chenfanfu Jiang, Yin Yang, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. GitHub

8. ElastoGen: 4D Generative Elastodynamics. Yutao Feng, Yintong Shang, Xiang Feng, Lei Lan, Shandian Zhe, Tianjia Shao, Hongzhi Wu, Kun Zhou, Hao Su, Chenfanfu Jiang, Yin Yang, arXiv, 2024. GitHub

9. NVFi: Neural Velocity Fields for 3D Physics Learning from Dynamic Videos. Jinxi Li, Ziyang Song, Bo Yang, Advances in Neural Information Processing Systems (NeurIPS), 2023. Paper

10. Inferring Hybrid Neural Fluid Fields from Videos. Hong-Xing Yu, Yang Zheng, Yuan Gao, Yitong Deng, Bo Zhu, Jiajun Wu, Advances in Neural Information Processing Systems (NeurIPS), 2023. Paper

11. NeuroFluid: Fluid Dynamics Grounding with Particle-Driven Neural Radiance Fields. Shanyan Guan, Huayu Deng, Yunbo Wang, Xiaokang Yang, International Conference on Machine Learning (ICML), 2022. Paper

12. Virtual Elastic Objects. Hsiao-yu Chen, Edgar Tretschk, Tuur Stuyck, Petr Kadlecek, Ladislav Kavan, Etienne Vouga, Christoph Lassner, arXiv, 2022. Paper

13. gradSim: Differentiable simulation for system identification and visuomotor control. Krishna Murthy Jatavallabhula, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine, Jerome Parent-Levesque, Kevin Xie, Kenny Erleben, Liam Paull, Florian Shkurti, Derek Nowrouzezahrai, Sanja Fidler, International Conference on Learning Representations (ICLR), 2021. Paper

14. One-Shot Real-to-Sim via End-to-End Differentiable Simulation and Rendering. Yifan Zhu, Tianyi Xiang, Aaron Dollar, Zherong Pan, arXiv, 2024. Paper

15. Physical Property Understanding from Language-Embedded Feature Fields. Albert J. Zhai, Yuan Shen, Emily Y. Chen, Gloria X. Wang, Xinlei Wang, Sheng Wang, Kaiyu Guan, Shenlong Wang, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. Paper

16. GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image. Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, Xiaoxiao Long, arXiv, 2024. GitHub

17. DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactions. Zhenjia Xu, Jiajun Wu, Andy Zeng, Joshua B. Tenenbaum, Shuran Song, arXiv, 2019. Paper

18. Visual Grounding of Learned Physical Models. Yunzhu Li, Toru Lin, Kexin Yi, Daniel M. Bear, Daniel L. K. Yamins, Jiajun Wu, Joshua B. Tenenbaum, Antonio Torralba, International Conference on Machine Learning (ICML), 2020. GitHub

19. Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids. Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, Antonio Torralba, arXiv, 2018. Paper

20. Physics 101: Learning Physical Object Properties from Unlabeled Videos. Jiajun Wu, Joseph J Lim, Hongyi Zhang, Joshua B Tenenbaum, William T Freeman, British Machine Vision Conference (BMVC), 2016. Paper

21. Interaction Networks for Learning about Objects, Relations and Physics. Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, Koray Kavukcuoglu, Advances in Neural Information Processing Systems (NeurIPS), 2016. Paper

22. Visual Vibrometry: Estimating Material Properties from Small Motions in Video. Abe Davis, Katherine L. Bouman, Justin G. Chen, Michael Rubinstein, Oral Büyüköztürk, Frédo Durand, William T. Freeman, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017. Paper

23. Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction. Vincent Le Guen, Nicolas Thome, Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2020. Paper

24. Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language. Mingyu Ding, Zhenfang Chen, Tao Du, Ping Luo, Joshua B. Tenenbaum, Chuang Gan, Advances In Neural Information Processing Systems (NeurIPS), 2021. GitHub

25. Flexible Neural Representation for Physics Prediction. Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li Fei-Fei, Joshua B Tenenbaum, Daniel L K Yamins, Advances in Neural Information Processing Systems (NeurIPS), 2018. Paper

26. Galileo: Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning. Jiajun Wu, Ilker Yildirim, Joseph J Lim, William T Freeman, Joshua B Tenenbaum Bcs, Advances in Neural Information Processing Systems (NeurIPS), 2015. Paper

27. GASP: Gaussian Splatting for Physic-Based Simulations. Piotr Borycki, Weronika Smolak, Joanna Waczyńska, Marcin Mazur, Sławomir Tadeja, Przemysław Spurek, arXiv, 2024. Paper

28. IntPhys 2019: A Benchmark for Visual Intuitive Physics Understanding. Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Veronique Izard, Emmanuel Dupoux, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. Paper

29. Learned Neural Physics Simulation for Articulated 3D Human Pose Reconstruction. Mykhaylo Andriluka, Baruch Tabanpour, C. Daniel Freeman, Cristian Sminchisescu, European Conference on Computer Vision (ECCV), 2024. Paper

30. Learning to See Physics via Visual De-animation. Jiajun Wu, Erika Lu, Pushmeet Kohli, William T Freeman, Joshua B Tenenbaum, Advances in Neural Information Processing Systems (NeurIPS), 2017. Paper

31. NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos. Yi-Ling Qiao, Alexander Gao, Ming C Lin, Advances in Neural Information Processing Systems, 2022. Paper

32. Neural Material Adaptor for Visual Grounding of Intrinsic Dynamics. Junyi Cao, Shanyan Guan, Yanhao Ge, Wei Li, Xiaokang Yang, Chao Ma, arXiv, 2024. Paper

33. Physical Representation Learning and Parameter Identification from Video Using Differentiable Physics. Rama Krishna Kandukuri, Jan Achterhold, Michael Moeller, Joerg Stueckler, International Journal of Computer Vision, 2022. Paper

34. Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video. Miguel Jaques, Michael Burke, Timothy Hospedales, arXiv, 2019. Paper

35. Unsupervised Learning for Physical Interaction through Video Prediction. Chelsea Finn, Ian Goodfellow Openai, Sergey Levine, Google Brain, Advances in Neural Information Processing Systems (NeurIPS), 2016. Paper

36. Visual Interaction Networks: Learning a Physics Simulator from Video. Nicholas Watters, Andrea Tacchetti, Théophane Weber, Razvan Pascanu, Peter Battaglia, Daniel Zoran, Advances in Neural Information Processing Systems (NeurIPS), 2017. Paper

37. Visual Physics: Discovering Physical Laws from Videos. Pradyumna Chari, Chinmay Talegaonkar, Yunhao Ba, Achuta Kadambi, arXiv, 2019. Paper

38. Learning to Identify Physical Parameters from Video Using Differentiable Physics. Rama Krishna Kandukuri, Jan Achterhold, Michael Möller, Jörg Stückler, arXiv, 2020. Paper

39. Scalable Differentiable Physics for Learning and Control. Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, Ming C. Lin, arXiv, 2020. Paper

40. PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos.. Meng Cao, Haoran Tang, Haoze Zhao, Hangyu Guo, Jiaheng Liu, Ge Zhang, Ruyang Liu, Qiang Sun, Ian Reid, and Xiaodan Liang, arXiv, 2024. Paper

41. MASS: Motion-Aware Spatial-Temporal Grounding for Physics Reasoning and Comprehension in Vision-Language Models.. Xiyang Wu, Zongxia Li, Jihui Jin, Guangyao Shi, Gouthaman KV, Vishnu Raj, Nilotpal Sinha, Jingxi Chen, Fan Du, and Dinesh Manocha, arXiv, 2025. Paper

42. CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Models.. Aaron Foss, Chloe Evans, Sasha Mitts, Koustuv Sinha, Ammar Rizvi, and Justine T. Kao, arXiv, 2025. Paper

43. Accelerating Physical Property Reasoning for Augmented Visual Cognition.. Hongbo Lan, Zhenlin An, Haoyu Li, Vaibhav Singh, and Longfei Shangguan, arXiv, 2025. Paper

Physics Evaluation

1. Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation. Fanqing Meng, Jiaqi Liao, Xinyu Tan, Wenqi Shao, Quanfeng Lu, Kaipeng Zhang, Yu Cheng, Dianqi Li, Yu Qiao, Ping Luo, arXiv, 2024. GitHub

2. GAIA: Rethinking Action Quality Assessment for AI-Generated Videos. Zijian Chen, Wei Sun, Yuan Tian, Jun Jia, Zicheng Zhang, Jiarui Wang, Ru Huang, Xiongkuo Min, Guangtao Zhai, Wenjun Zhang, arXiv, 2024. GitHub

3. MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions. Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, Ying Shan, arXiv, 2024. GitHub

4. Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification. S. P. Sharan, Minkyu Choi, Sahil Shah, Harsh Goel, Mohammad Omama, Sandeep Chinchali, arXiv, 2024. Paper

5. Quality Prediction of AI Generated Images and Videos: Emerging Trends and Opportunities. Abhijay Ghildyal, Yuanhan Chen, Saman Zadtootaghaj, Nabajeet Barman, Alan C. Bovik, arXiv, 2024. Paper

6. T2VBench: Benchmarking Temporal Dynamics for Text-to-Video Generation. Pengliang Ji, Chuyang Xiao, Huilin Tai, Mingxiao Huo, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. Paper

7. TlTScore: Towards Long-Tail Effects in Text-to-Visual Evaluation with Generative Foundation Models. Pengliang Ji, Junchen Liu, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. Paper

8. VBench: Comprehensive Benchmark Suite for Video Generative Models. Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, Ziwei Liu, arXiv, 2023. Homepage

9. VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models. Ziqi Huang, Fan Zhang, Xiaojie Xu, Yinan He, Jiashuo Yu, Ziyue Dong, Qianli Ma, Nattapol Chanpaisit, Chenyang Si, Yuming Jiang, Yaohui Wang, Xinyuan Chen, Ying-Cong Chen, Limin Wang, Dahua Lin, Yu Qiao, Ziwei Liu, arXiv, 2024. GitHub

10. VideoPhy: Evaluating Physical Commonsense for Video Generation. Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, Aditya Grover, arXiv, 2024. GitHub

11. VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation. Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, Kai Wang, Quy Duc Do, Yuansheng Ni, Bohan Lyu, Yaswanth Narsupalli, Rongqi Fan, Zhiheng Lyu, Yuchen Lin, Wenhu Chen, arXiv, 2024. Homepage

12. What You See Is What Matters: A Novel Visual and Physics-Based Metric for Evaluating Video Generation Quality. Zihan Wang, Songlin Li, Lingyan Hao, Xinyu Hu, Bowen Song, arXiv, 2024. Paper

13. WorldSimBench: Towards Video Generation Models as World Simulators. Yiran Qin, Zhelun Shi, Jiwen Yu, Xijun Wang, Enshen Zhou, Lijun Li, Zhenfei Yin, Xihui Liu, Lu Sheng, Jing Shao, Lei Bai, Wanli Ouyang, Ruimao Zhang, arXiv, 2024. GitHub

14. PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Models. Fanqing Meng, Wenqi Shao, Lixin Luo, Yahong Wang, Yiran Chen, Quanfeng Lu, Yue Yang, Tianshuo Yang, Kaipeng Zhang, Yu Qiao, Ping Luo, arXiv, 2024. Paper

15. T2I-FactualBench: Benchmarking the Factuality of Text-to-Image Models with Knowledge-Intensive Concepts. Ziwei Huang, Wanggui He, Quanyu Long, Yandi Wang, Haoyuan Li, Zhelun Yu, Fangxun Shu, Long Chan, Hao Jiang, Leilei Gan, Fei Wu, arXiv, 2024. Paper

16. ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation. Shenghai Yuan, Jinfa Huang, Yongqi Xu, Yaoyang Liu, Shaofeng Zhang, Yujun Shi, Ruijie Zhu, Xinhua Cheng, Jiebo Luo, Li Yuan, NeurIPS D&B Spotlight, 2024. Paper Github Homepage

17. Evaluating Text-to-Visual Generation with Image-to-Text Generation. Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, Deva Ramanan, European Conference on Computer Vision (ECCV), 2024. Homepage

18. GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation.. Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, Deva Ramanan, arXiv, 2024. Homepage

19. 4DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Models.. Yiting Lu, Wei Luo, Peiyan Tu, Haoran Li, Hanxin Zhu, Zihao Yu, Xingrui Wang et al., arXiv, 2025. Homepage

20. Impossible videos.. Zechen Bai, Hai Ci, and Mike Zheng Shou, arXiv, 2025. Paper

21. InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoning.. Gautam Sreekumar, and Vishnu Naresh Boddeti, arXiv, 2025. Paper

22. IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments.. Florian Bordes, Quentin Garrido, Justine T. Kao, Adina Williams, Michael Rabbat, and Emmanuel Dupoux, arXiv, 2025. Paper

23. Intuitive Physics Understanding Emerges from Self-Supervised Pretraining on Natural Videos.. Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, and Yann LeCun, arXiv, 2025. Paper

24. LikePhys: Evaluating Intuitive Physics Understanding in Video Diffusion Models via Likelihood Preference.. Jianhao Yuan, Fabio Pizzati, Francesco Pinto, Lars Kunze, Ivan Laptev, Paul Newman, Philip Torr, and Daniele De Martini, arXiv, 2025. Paper

25. Morpheus: Benchmarking Physical Reasoning of Video Generative Models with Real Physical Experiments.. Chenyu Zhang, Daniil Cherniavskii, Antonios Tragoudaras, Antonios Vozikis, Thijmen Nijdam, Derck WE Prinzhorn, Mark Bodracska, Nicu Sebe, Andrii Zadaianchuk, and Efstratios Gavves, arXiv, 2025. Paper

26. PAI-Bench: A Comprehensive Benchmark For Physical AI.. Fengzhe Zhou, Jiannan Huang, Jialuo Li, Deva Ramanan, and Humphrey Shi, arXiv, 2025. Paper

27. PhyDetEx: Detecting and Explaining the Physical Plausibility of T2V Models.. Zeqing Wang, Keze Wang, and Lei Zhang, arXiv, 2025. Paper

28. PhyWorldBench: A Comprehensive Evaluation of Physical Realism in Text-to-Video Models.. Jing Gu, Xian Liu, Yu Zeng, Ashwin Nagarajan, Fangrui Zhu, Daniel Hong, Yue Fan et al., arXiv, 2025. Paper

29. Science-T2I: Addressing Scientific Illusions in Image Synthesis.. Jialuo Li, Wenhao Chai, Xingyu Fu, Haiyang Xu, and Saining Xie, In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), 2025. Paper

30. Spotlight: Identifying and Localizing Video Generation Errors Using VLMs.. Aditya Chinchure, Sahithya Ravi, Pushkar Shukla, Vered Shwartz, and Leonid Sigal, arXiv, 2025. Paper

31. T2VPhysBench: A First-Principles Benchmark for Physical Consistency in Text-to-Video Generation. Xuyang Guo, Jiayan Huo, Zhenmei Shi, Zhao Song, Jiahao Zhang, and Jiale Zhao, arXiv, 2025. Paper

32. VideoVerse: How Far is Your T2V Generator from a World Model?.. Zeqing Wang, Xinyu Wei, Bairui Li, Zhen Guo, Jinrui Zhang, Hongyang Wei, Keze Wang, and Lei Zhang, arXiv, 2025. Paper

33. WorldModelBench: Judging Video Generation Models as World Models.. Dacheng Li, Yunhao Fang, Yukang Chen, Shuo Yang, Shiyi Cao, Justin Wong, Michael Luo et al, arXiv, 2025. Homepage

34. WorldScore: A Unified Evaluation Benchmark for World Generation.. Haoyi Duan, Hong-Xing Yu, Sirui Chen, Li Fei-Fei, and Jiajun Wu, arXiv, 2025. Homepage