Official implementation for the paper "Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI"
See the codeREADME
Table of Contents generated with DocToc
- Awesome AI+Physics
Awesome AI+Physics
A curated union of AI-for-physics literature, organized with the taxonomy from Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI.
1. Physical Perception
1.1 Object Recognition
- An expert ensemble for detecting anomalous scenes, interactions, and behaviors in autonomous driving — Int. J. Robotics Res., 2025.
- Qwen3-Omni technical report — arXiv:2509.17765, 2025.
- Towards visual discrimination and reasoning of real-world physical dynamics: Physics-grounded anomaly detection — CVPR, 2025.
- Cognition guided video anomaly detection framework for surveillance services — TSC, 2024.
- Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection — ECCV, 2024.
- Inter-object discriminative graph modeling for indoor scene recognition — KBS, 2024.
- View-invariant pixelwise anomaly detection in multi-object scenes with adaptive view synthesis — arXiv:2406.18012, 2024.
- BOP Challenge 2020 on 6D object localization — Eur. Conf. Comput. Vis. Workshops, 2020.
- Mask R-CNN — Proc. IEEE Int. Conf. Comput. Vis., 2017.
- Microsoft COCO: Common objects in context — ECCV, 2014.
- ImageNet: A large-scale hierarchical image database — CVPR, 2009.
1.2 Spatial Perception
- MMSI-Bench: A benchmark for multi-image spatial intelligence — ICLR, 2026.
- Contrastive learning of image representations guided by spatial relations — Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis., 2025.
- Multi-SpatialMLLM: multi-frame spatial understanding with multi-modal large language models — arXiv:2505.17015, 2025.
- Open3DVQA: A benchmark for comprehensive spatial reasoning with multimodal large language model in open space — arXiv:2503.11094, 2025.
- Spatial-MLLM: Boosting MLLM capabilities in visual-based spatial intelligence — arXiv:2505.23747, 2025.
- SpatialScore: Towards unified evaluation for multimodal spatial understanding — arXiv:2505.17012, 2025.
- ViewSpatial-Bench: Evaluating multi-perspective spatial localization in vision-language models — arXiv:2505.21500, 2025.
- Comprehensive multi-view representation learning — Inf. Fusion, 2023.
1.3 Identifying Intrinsic Property
- PhysID: Physics-based interactive dynamics from a single-view image — IEEE Int. Conf. Acoust., Speech, Signal Process., 2025.
- AI-based density recognition — WSCG, 2024.
- Intrinsic image diffusion for indoor single-view material estimation — CVPR, 2024.
- Practical measurements of translucent materials with inter-pixel translucency prior — CVPR, 2024.
- Vision-based approach for food weight estimation from 2D images — arXiv:2405.16478, 2024.
- Fabric surface characterization: assessment of deep learning-based texture representations using a challenging dataset — J. Text. Inst., 2021.
1.4 Dynamic Estimation
- PhysID: Physics-based interactive dynamics from a single-view image — IEEE Int. Conf. Acoust., Speech, Signal Process., 2025.
- Reasoning-enhanced object-centric learning for videos — Proc. ACM SIGKDD Conf. Knowl. Discov. Data Min., 2025.
- ContPhy: Continuum physical concept learning and reasoning from videos — ICML, 2024.
- I-PHYRE: Interactive physical reasoning — ICLR, 2024.
- Physion++: Evaluating physical scene understanding that requires online inference of different physical properties — NeurIPS, 2023.
- SlotFormer: Unsupervised visual dynamics simulation with object-centric models — ICLR, 2023.
- Physion: Evaluating physical prediction from vision in humans and machines — NeurIPS, 2021.
- A compositional object-based approach to learning physical dynamics — ICLR, 2017.
- Visual interaction networks: Learning a physics simulator from video — NIPS, 2017.
- Interaction networks for learning about objects, relations and physics — NIPS, 2016.
- Newtonian scene understanding: Unfolding the dynamics of objects in static images — CVPR, 2016.
2. Physics Reasoning
2.1 Symbolic Reasoning
- ABench-Physics: Benchmarking physical reasoning in LLMs via high-difficulty and dynamic physics problems — arXiv:2507.04766, 2025.
- LOCA-R: Near-Perfect Performance on the Chinese Physics Olympiad 2025 — arXiv:2511.10515, 2025.
- PHYBench: Holistic evaluation of physical perception and reasoning in large language models — arXiv:2504.16074, 2025.
- Physics Supernova: AI agent matches elite gold medalists at IPhO 2025 — arXiv:2509.01659, 2025.
- PhysicsEval: inference-time techniques to improve the reasoning proficiency of large language models on physics problems — Findings of IJCNLP, 2025.
- Symbolic or numerical? Understanding physics problem solving in reasoning LLMs — arXiv:2507.01334, 2025.
- UGPhysics: A comprehensive benchmark for undergraduate physics reasoning with large language models — arXiv:2502.00334, 2025.
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark — COLM, 2024.
- Knowledge graphs are all you need: Leveraging KGs in physics question answering — arXiv:2412.05453, 2024.
- LLMPhy: Complex physical reasoning using large language models and world models — arXiv:2411.08027, 2024.
- OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems — ACL, 2024.
- Steps are all you need: Rethinking STEM education with prompt engineering — arXiv:2412.05023, 2024.
- Could an artificial-intelligence agent pass an introductory physics course? — Phys. Rev. Phys. Educ. Res., 2023.
- Chain-of-thought prompting elicits reasoning in large language models — NeurIPS, 2022.
2.2 Multimodal-grounded Reasoning
- P1-VL: bridging visual perception and scientific reasoning in physics Olympiads — arXiv:2602.09443, 2026.
- TRACE: A framework for analyzing and enhancing stepwise reasoning in vision-language models — EACL, 2026.
- DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning — Nat., 2025.
- MMMU-Pro: A more robust multi-discipline multimodal understanding benchmark — ACL, 2025.
- Multimodal reasoning for science: Technical report and 1st place solution to the ICML 2025 SeePhys challenge — arXiv:2509.06079, 2025.
- PhysReason: A comprehensive benchmark towards physics-based reasoning — ACL, 2025.
- SeePhys: Does seeing help thinking?–Benchmarking vision-based physics reasoning — arXiv:2505.19099, 2025.
- MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI — CVPR, 2024.
- ComPhy: Compositional physical reasoning of objects and events from videos — ICLR, 2022.
- Grounding physical concepts of objects and events through dynamic visual reasoning — ICLR, 2021.
- CLEVRER: Collision events for video representation and reasoning — ICLR, 2020.
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
2.3 Causal and Counterfactual Reasoning
- CAUSAL3D: A comprehensive benchmark for causal learning from visual data — arXiv:2503.04852, 2025.
- PhySense: Principle-based physics reasoning benchmarking for large language models — arXiv:2505.24823, 2025.
- Using causal threads to explain changes in a dynamic system — Int. Conf. Asian Digit. Libr., 2023.
- CLEVRER-Humans: Describing physical and causal events the human way — NeurIPS, 2022.
- Dynamic visual reasoning by learning differentiable physics models from video and language — NeurIPS, 2021.
- Learning neural causal models from unknown interventions — arXiv:1910.01075, 2020.
- Causal modeling of dynamical systems — arXiv:1803.08784, 2018.
- GRASP: A novel benchmark for evaluating language GRounding And Situated Physics understanding in multimodal language models
- IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments
- PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding
2.4 Accelerate Physics Research
2.4.1 Hypothesis Generation
- ASP-assisted symbolic regression: Uncovering hidden physics in fluid mechanics — arXiv:2507.17777, 2025.
- Interpretable data-driven turbulence modeling for separated flows using symbolic regression with unit constraints — Acta Mech., 2025.
- Parsing the language of expression: Enhancing symbolic regression with domain-aware symbolic priors — arXiv:2503.09592, 2025.
- SRBench++: Principled benchmarking of symbolic regression with domain-expert interpretation — TEVC, 2025.
- A multi-agent framework for physical laws discovery — arXiv:2411.16416, 2024.
- Combining genetic algorithm and compressed sensing for features and operators selection in symbolic regression — arXiv:2403.15816, 2024.
- Contemporary symbolic regression methods and their relative performance — NeurIPS D B, 2021.
- AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity
- AI Feynman: A physics-inspired method for symbolic regression
- AI-Newton: A Concept-Driven Physical Law Discovery System without Prior Physical Knowledge
- Bayesian symbolic regression: Automated equation discovery from a physicists' perspective
- Discovering physical concepts with neural networks
- Intelligence, physics and information – the tradeoff between accuracy and simplicity in machine learning
- KAN 2.0: Kolmogorov-Arnold Networks Meet Science
- KAN: Kolmogorov-Arnold Networks
- Toward an AI Physicist for Unsupervised Learning
2.4.2 Streamline Experimentation
- Autonomous agents coordinating distributed discovery through emergent artifact exchange — arXiv:2603.14312, 2026.
- From agent-only social networks to autonomous scientific research: Lessons from OpenClaw and Moltbook, and the architecture of ClawdLab and Beach.Science — arXiv:2602.19810, 2026.
- Large language model driven development of turbulence models — Flow, 2025.
- Open source planning & control system with language agents for autonomous scientific discovery — ICML Workshop, 2025.
- ScienceAgentBench: Toward rigorous assessment of language agents for data-driven scientific discovery — ICLR, 2025.
- The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search — arXiv:2504.08066, 2025.
- AtomAgents: Alloy design and discovery through physics-aware multi-modal multi-agent artificial intelligence — arXiv:2407.10022, 2024.
- SciCode: A research coding benchmark curated by scientists — NeurIPS, 2024.
- The AI scientist: Towards fully automated open-ended scientific discovery — arXiv:2408.06292, 2024.
3. World Modeling
3.1 Image Generation
- Close the optical sensing domain gap by physics-grounded active stereo sensor simulation — TRO, 2023.
- ENVIDR: Implicit differentiable renderer with neural environment lighting — ICCV, 2023.
- Orbit: A unified simulation framework for interactive robot learning environments — RA-L, 2023.
- Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields — CVPR, 2022.
- CLIPScore: A reference-free evaluation metric for image captioning — EMNLP, 2021.
- Unity: A general platform for intelligent agents — arXiv:1809.02627, 2018.
- GANs trained by a two time-scale update rule converge to a local Nash equilibrium — NIPS, 2017.
- Indoor segmentation and support inference from RGBD images — ECCV, 2012.
- A reflectance model for computer graphics — TOG, 1982.
- Illumination for computer generated pictures — Commun. ACM, 1975.
- Generative AI for Validating Physics Laws
- PBR-NeRF: Inverse Rendering with Physics-Based Neural Fields
- Progressive Growing of GANs for Improved Quality, Stability, and Variation
3.2 Video Generation
- CogVideoX: text-to-video diffusion models with an expert transformer — ICLR, 2025.
- DINO-Foresight: Looking into the future with DINO — NeurIPS, 2025.
- Do generative video models learn physical principles from watching videos? — arXiv:2501.09038, 2025.
- LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models — IJCV, 2025.
- Morpheus: Benchmarking physical reasoning of video generative models with real physical experiments — arXiv:2504.02918, 2025.
- ProPhy: Progressive Physical Alignment for Dynamic World Simulation — arXiv:2512.05564, 2025.
- Think before you diffuse: LLMs-guided physics-aware video generation — arXiv:2505.21653, 2025.
- Towards world simulator: Crafting physical commonsense-based benchmark for video generation — ICML, 2025.
- VideoREPA: Learning physics for video generation through relational alignment with foundation models — NeurIPS, 2025.
- VLIPP: Towards physically plausible video generation with vision and language informed physical prior — ICCV, 2025.
- WorldModelBench: Judging video generation models as world models — arXiv:2502.20694, 2025.
- GPT4Motion: Scripting physical motions in text-to-video generation via Blender-oriented GPT planning — Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops, 2024.
- MotionCraft: Physics-Based zero-shot video generation — NeurIPS, 2024.
- Open-Sora: Democratizing efficient video production for all — arXiv:2412.20404, 2024.
- PhysGen: Rigid-Body physics-grounded image-to-video generation — ECCV, 2024.
- VideoPhy: Evaluating physical commonsense for video generation — arXiv:2406.03520, 2024.
- GAIA-1: A generative world model for autonomous driving — arXiv:2309.17080, 2023.
- VideoPoet: A large language model for zero-shot video generation — arXiv:2312.14125, 2023.
- IntrinsicAvatar: Physically Based Inverse Rendering of Dynamic Humans from Monocular Videos via Explicit Ray Tracing
- Self-Supervised Visual Planning with Temporal Skip Connections
- T2VPhysBench: A First-Principles Benchmark for Physical Consistency in Text-to-Video Generation
- The "Something Something" Video Database for Learning and Evaluating Visual Common Sense
- The Kinetics Human Action Video Dataset
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Unsupervised Learning of Video Representations using LSTMs
- VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation
3.3 Scene Reconstruction
- DriveDreamer4D: World models are effective data machines for 4D driving scene representation — CVPR, 2025.
- ReconDreamer: Crafting world models for driving scene reconstruction via online restoration — CVPR, 2025.
- PhyRecon: Physically plausible neural scene reconstruction — NeurIPS, 2024.
- Multiview neural surface reconstruction by disentangling geometry and appearance — NeurIPS, 2020.
- A multi-view stereo benchmark with high-resolution images and multi-camera videos — CVPR, 2017.
- GANs trained by a two time-scale update rule converge to a local Nash equilibrium — NIPS, 2017.
- Tanks and Temples: Benchmarking large-scale scene reconstruction — TOG, 2017.
- Large scale multi-view stereopsis evaluation — CVPR, 2014.
- MuJoCo: A physics engine for model-based control — IROS, 2012.
- A comparison and evaluation of multi-view stereo reconstruction algorithms — Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2006.
- The interpretation of structure from motion — Proc. R. Soc. Lond. B, Biol. Sci., 1979.
- PBR-NeRF: Inverse Rendering with Physics-Based Neural Fields
3.4 Physics-constrained Simulation
- PhyWorldBench: A comprehensive evaluation of physical realism in text-to-video models — ICLR, 2026.
- Back to the features: DINO as a foundation for video world models — arXiv:2507.19468, 2025.
- DEM-NeRF: A neuro-symbolic method for scientific discovery through physics-informed simulation — arXiv:2507.21350, 2025.
- FusionForce: End-to-end differentiable neural-symbolic layer for trajectory prediction — arXiv:2502.10156, 2025.
- How far is video generation from world model: A physical law perspective — ICML, 2025.
- Neural motion simulator: Pushing the limit of world models in reinforcement learning — CVPR, 2025.
- NeoPhysIx: An ultra fast 3D physical simulator as development tool for AI algorithms — arXiv:2411.05799, 2024.
- PhysORD: a neuro-symbolic approach for physics-infused motion prediction in off-road driving — IROS, 2024.
- Brax – A differentiable physics engine for large scale rigid body simulation — NeurIPS, 2021.
- LagNetViP: A Lagrangian neural network for video prediction — AAAI, 2020.
- Combining physical simulators and object-based networks for control — ICRA, 2019.
- PHYRE: A new benchmark for physical reasoning — NeurIPS, 2019.
- MuJoCo: A physics engine for model-based control — IROS, 2012.
- Brax: Massively Parallel Rigidbody Physics Simulation on Accelerator Hardware
- Building World Models with Neural Physics
- Isaac Sim: Robotics Simulation and Synthetic Data Generation
- MuJoCo: Advanced Physics Simulation
- PyBullet: Time Physics Simulation
4. Embodied Interaction
4.1 Robotics
- DriveVA: Video Action Models are Zero-Shot Drivers — arXiv:2604.04198, 2026.
- World Action Models are Zero-shot Policies — arXiv:2602.15922, 2026.
- World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetry — arXiv:2604.01985, 2026.
- Diffusion policy: Visuomotor policy learning via action diffusion — IJRR, 2025.
- EmbodiedBench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents — ICML, 2025.
- EMMOE: A comprehensive benchmark for embodied mobile manipulation in open environments — arXiv:2503.08604, 2025.
- OpenVLA: An open-source vision-language-action model — CoRL, 2025.
- Vision-language-action models: Concepts, progress, applications and challenges — arXiv:2505.04769, 2025.
- Active exploration for contact-rich tasks via online-adaptable models — ICRA, 2024.
- Open X-Embodiment: Robotic learning datasets and RT-X models — ICRA, 2024.
- π_0: A vision-language-action flow model for general robot control — arXiv:2410.24164, 2024.
- Active visuo-tactile interactive robotic perception for object property-aware manipulation — arXiv:2310.15551, 2023.
- Flow Matching for Generative Modeling — ICLR, 2023.
- ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills — International Conference on Learning Representations (ICLR), 2023.
- RT-1: Robotics Transformer for real-world control at scale — RSS, 2023.
- RT-2: Vision-language-action models transfer web knowledge to robotic control — CoRL, 2023.
- A generalist agent — Trans. Mach. Learn. Res., 2022.
- CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks — 2022.
- Do as I can, not as I say: Grounding language in robotic affordances — CoRL, 2022.
- Safety-critical kinematic control of robotic systems — IEEE Control Syst. Lett., 2022.
- BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments — 2021.
- BOP Challenge 2020 on 6D object localization — Eur. Conf. Comput. Vis. Workshops, 2020.
- GraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping — CVPR, 2020.
- RLBench: The Robot Learning Benchmark & Learning Environment — IEEE Robotics and Automation Letters, 2020.
- robosuite: A Modular Simulation Framework and Benchmark for Robot Learning — 2020.
- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning — 2019.
4.2 Navigation
- CL-CoTNav: Closed-loop hierarchical chain-of-thought for zero-shot object-goal navigation with vision-language models — arXiv:2504.09000, 2025.
- EmbodiedBench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents — ICML, 2025.
- FLAME: Learning to navigate with multimodal LLM in urban environments — AAAI, 2025.
- LGR: LLM-guided ranking of frontiers for object goal navigation — arXiv:2503.20241, 2025.
- NavBench: A unified robotics benchmark for reinforcement learning-based autonomous navigation — arXiv:2505.14526, 2025.
- NavCoT: Boosting LLM-based vision-and-language navigation via learning disentangled reasoning — TPAMI, 2025.
- NaVILA: Legged robot vision-language-action model for navigation — RSS, 2025.
- Research on navigation methods based on LLMs — arXiv:2504.15600, 2025.
- Stairway to success: zero-shot floor-aware object-goal navigation via LLM-driven coarse-to-fine exploration — arXiv:2505.23019, 2025.
- Uncovering the fragility of trustworthy LLMs through Chinese textual ambiguity — Agentic GenAI Evaluation Workshop KDD'25, 2025.
- X-MOBILITY: End-To-End Generalizable Navigation Via World Modeling — ICRA, 2025.
- Advancing object goal navigation through LLM-enhanced object affinities transfer — arXiv:2403.09971, 2024.
- Can LLM be a good path planner based on prompt engineering? Mitigating the hallucination for path planning — arXiv:2408.13184, 2024.
- HM3D-OVON: A dataset and benchmark for open-vocabulary object goal navigation — IROS, 2024.
- NavGPT: Explicit reasoning in vision-and-language navigation with large language models — AAAI, 2024.
- TWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfer — ICRA, 2024.
- UNMuTe: Unifying navigation and multimodal dialogue-like text generation — arXiv:2408.04423, 2024.
- VELMA: Verbalization embodiment of LLM agents for vision and language navigation in Street View — AAAI, 2024.
- Vision-and-language navigation today and tomorrow: A survey in the era of foundation models — arXiv:2407.07035, 2024.
- Object goal navigation with recursive implicit maps — IROS, 2023.
- ADAPT: Vision-language navigation with modality-aligned action prompts — CVPR, 2022.
- Chain-of-thought prompting elicits reasoning in large language models — NeurIPS, 2022.
- Goal-oriented vision-and-dialog navigation via reinforcement learning — Findings of EMNLP, 2022.
- Think global, act local: Dual-scale graph transformer for vision-and-language navigation — CVPR, 2022.
- History aware multimodal transformer for vision-and-language navigation — NeurIPS, 2021.
- The RobotSlang benchmark: Dialog-guided robot localization and navigation — CoRL, 2021.
- BabyWalk: Going farther in vision-and-language navigation by taking baby steps — Proc. 58th Annu. Meet. Assoc. Comput. Linguist., 2020.
- Beyond the nav-graph: Vision-and-language navigation in continuous environments — ECCV, 2020.
- Interactive Gibson benchmark: A benchmark for interactive navigation in cluttered environments — RA-L, 2020.
- Object goal navigation using goal-oriented semantic exploration — NeurIPS, 2020.
- REVERIE: Remote embodied visual referring expression in real indoor environments — CVPR, 2020.
- RoboTHOR: An open simulation-to-real embodied AI platform — CVPR, 2020.
- Room-Across-Room: Multilingual vision-and-language navigation with dense spatiotemporal grounding — Proc. 2020 Conf. Empir. Methods Nat. Lang. Process., 2020.
- Vision-and-dialog navigation — CoRL, 2020.
- Vision-dialog navigation by exploring cross-modal memory — CVPR, 2020.
- Learning to navigate unseen environments: Back translation with environmental dropout — Proc. Conf. North Amer. Chapter Assoc. Comput. Linguistics: Hum. Lang. Technol., 2019.
- TOUCHDOWN: Natural language navigation and spatial reasoning in visual street environments — CVPR, 2019.
- Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments — CVPR, 2018.
- AI2-THOR: An Interactive 3D Environment for Visual AI — arXiv:1712.05474, 2017.
- LLM ContextBridge: A Hybrid Approach for Intent and Dialogue Understanding in IVSR
- Matterport3D: Learning from RGB-D Data in Indoor Environments
- R2H: Building Multimodal Navigation Helpers that Respond to Help Requests
- Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation
4.3 Autonomous Driving
- AlphaDrive: Unleashing the power of VLMs in autonomous driving via reinforcement learning and reasoning — arXiv:2503.07608, 2025.
- AutoDrive-R^2: Incentivizing reasoning and self-reflection capacity for VLA model in autonomous driving — arXiv:2509.01944, 2025.
- Cosmos-Drive-Dreams: Scalable synthetic driving data generation with world foundation models — arXiv:2506.09042, 2025.
- DriveLMM-o1: A step-by-step reasoning dataset and large multimodal model for driving scenario understanding — arXiv:2503.10621, 2025.
- Driving in the occupancy world: Vision-centric 4D occupancy forecasting and planning via world models for autonomous driving — AAAI, 2025.
- DrivingGPT: Unifying driving world modeling and planning with multi-modal autoregressive transformers — ICCV, 2025.
- LeapVAD: A leap in autonomous driving via cognitive perception and dual-process thinking — TNNLS, 2025.
- Mastering diverse control tasks through world models — Nature, 2025.
- PRIMEDrive-CoT: A Precognitive Chain-of-Thought Framework for Uncertainty-Aware Object Interaction in Driving Scene Scenario — Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops, 2025.
- S2R-Bench: A sim-to-real evaluation benchmark for autonomous driving — Sci. Data, 2025.
- TeraSim-World: Worldwide safety-critical data synthesis for end-to-end autonomous driving — arXiv:2509.13164, 2025.
- Bench2Drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving — NeurIPS, 2024.
- DriveCoT: Integrating chain-of-thought reasoning with end-to-end driving — arXiv:2403.16996, 2024.
- DriveDreamer: Towards real-world-driven world models for autonomous driving — ECCV, 2024.
- OccLLaMA: An occupancy-language-action generative world model for autonomous driving — arXiv:2409.03272, 2024.
- Reason2Drive: Towards interpretable and chain-based reasoning for autonomous driving — ECCV, 2024.
- Think2Drive: Efficient reinforcement learning by thinking with latent world model for autonomous driving (in CARLA-v2) — ECCV, 2024.
- A survey on safety-critical driving scenario generation—A methodological perspective — TITS, 2023.
- Interaction-aware trajectory planning for autonomous vehicles with analytic integration of neural networks into model predictive control — ICRA, 2023.
- Deep reinforcement learning for autonomous driving: A survey — TITS, 2022.
- MotionSC: data set and network for real-time semantic mapping in dynamic environments — RA-L, 2022.
- Wayformer: Motion forecasting via simple & efficient attention networks — arXiv:2207.05844, 2022.
- NuPlan: a closed-loop ML-based planning benchmark for autonomous vehicles — arXiv:2106.11810, 2021.
- A survey of deep learning techniques for autonomous driving — J. Field Robot., 2020.
- Human motion trajectory prediction: a survey — IJRR, 2020.
- Learning lane graph representations for motion forecasting — ECCV, 2020.
- nuScenes: A multimodal dataset for autonomous driving — CVPR, 2020.
- Scalability in perception for autonomous driving: Waymo open dataset — CVPR, 2020.
- ChauffeurNet: Learning to drive by imitating the best and synthesizing the worst — RSS, 2019.
- A framework for automated driving system testable cases and scenarios — 2018.
- AirSim: High-fidelity visual and physical simulation for autonomous vehicles — Field Serv. Robot., 2018.
- World models — arXiv:1803.10122, 2018.
- CARLA: An open urban driving simulator — CoRL, 2017.
- CommonRoad: Composable benchmarks for motion planning on roads — IEEE Intell. Veh. Symp., 2017.
- A survey of motion planning and control techniques for self-driving urban vehicles — IEEE Trans. Intell. Veh., 2016.
- End to end learning for self-driving cars — arXiv:1604.07316, 2016.
- Maximum entropy inverse reinforcement learning — AAAI, 2008.
- A survey of industrial model predictive control technology — Control Eng. Pract., 2003.
- Congested traffic states in empirical observations and microscopic simulations — Phys. Rev. E, 2000.
- Rapidly-exploring random trees: A new tool for path planning — Research Report 9811, 1998.
- Real-time obstacle avoidance for manipulators and mobile robots — IJRR, 1986.
- Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting
- Argoverse: 3D Tracking and Forecasting with Rich Maps
- BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning
- BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layout
- Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion
- DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Model
- DreamDrive: Generative 4D Scene Modeling from Street View Images
- DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model
- DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT
- HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation
- HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving
- Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving
- OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving
- One Thousand and One Hours: Self-driving Motion Prediction Dataset
- RenderWorld: World Model with Self-Supervised 3D Label
- RoboTron-Sim: Improving Real-World Driving via Simulated Hard-Case
- Stag-1: Towards Realistic 4D Driving Simulation with Video Generation Model
- The ApolloScape Open Dataset for Autonomous Driving and its Application
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)
- UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generation
- Vision meets robotics: The KITTI dataset
- Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability
5. Survey-wide Foundations and Repository Extensions
5.1 Survey-wide Foundations and Cross-cutting References
- From System 1 to System 2: A survey of reasoning large language models — TPAMI, 2026.
- Object detection with multimodal large vision-language models: An in-depth review — Inf. Fusion, 2026.
- The artificial intelligence cognitive examination: A survey on the evolution of multimodal evaluation from recognition to reasoning — IEEE Access, 2026.
- 3D and 4D world modeling: A survey — arXiv:2509.07996, 2025.
- A physics-guided multimodal transformer path to weather and climate sciences — arXiv:2504.14174, 2025.
- A survey of embodied learning for object-centric robotic manipulation — Mach. Intell. Res., 2025.
- A survey on world models grounded in acoustic physical information — arXiv:2506.13833, 2025.
- A survey: Learning embodied intelligence from physical simulators and world models — arXiv:2507.00917, 2025.
- Embodied AI: From LLMs to world models — IEEE Circuits Syst. Mag., 2025.
- Foundation model driven robotics: A comprehensive review — arXiv:2507.10087, 2025.
- Foundation models for autonomous driving perception: A survey through core capabilities — arXiv:2509.08302, 2025.
- From 2D to 3D cognition: A brief survey of general world models — arXiv:2506.20134, 2025.
- From perception to cognition: A survey of vision-language interactive reasoning in multimodal large language models — arXiv:2509.25373, 2025.
- Generative physical AI in vision: A survey — arXiv:2501.10928, 2025.
- Large model empowered embodied AI: A survey on decision-making and embodied learning — arXiv:2508.10399, 2025.
- Large physics models: towards a collaborative approach with large language models and foundation models — EPJ C, 2025.
- MINT-CoT: Enabling interleaved visual tokens in mathematical chain-of-thought reasoning — arXiv:2506.05331, 2025.
- PhyBlock: A progressive benchmark for physical understanding and planning via 3D block assembly — NeurIPS, 2025.
- Probing perceptual constancy in large vision language models — arXiv:2502.10273, 2025.
- Qwen2.5-VL technical report — arXiv:2502.13923, 2025.
- Reinforcement learning meets large language models: A survey of advancements and applications across the LLM lifecycle — arXiv:2509.16679, 2025.
- Toward embodied AGI: A review of embodied AI and the road ahead — arXiv:2505.14235, 2025.
- Understanding world or predicting future? A comprehensive survey of world models — CSUR, 2025.
- A survey on large language model based autonomous agents — Front. Comput. Sci., 2024.
- A survey on multimodal large language models — NSR, 2024.
- A survey on robotics with foundation models: toward embodied AI — arXiv:2402.02385, 2024.
- AI meets physics: a comprehensive survey — Artif. Intell. Rev., 2024.
- Aligning cyber space with physical world: A comprehensive survey on embodied AI — arXiv:2407.06886, 2024.
- From efficient multimodal models to world models: A survey — arXiv:2407.00118, 2024.
- GPT-4o system card — arXiv:2410.21276, 2024.
- Is Sora a world simulator? A comprehensive survey on general world models and beyond — arXiv:2405.03520, 2024.
- OpenAI o1 system card — arXiv:2412.16720, 2024.
- PhyGrasp: generalizing robotic grasping with physics-informed large multimodal models — arXiv:2402.16836, 2024.
- ShapeLLM: Universal 3D object understanding for embodied interaction — ECCV, 2024.
- A survey on deep reinforcement learning algorithms for robotic manipulation — Sensors, 2023.
- Towards reasoning in large language models: A survey — Findings of ACL, 2023.
- Visual instruction tuning — NeurIPS, 2023.
- A survey of embodied AI: From simulators to research tasks — TETCI, 2022.
- A survey on machine learning approaches for modelling intuitive physics — IJCAI, 2022.
- Functional optimization of fluidic devices with differentiable Stokes flow — TOG, 2020.
- Scalable differentiable physics for learning and control — ICML, 2020.
- Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data — CMAME, 2020.
- Machine learning for data-driven discovery in solid Earth geoscience — Science, 2019.
- Solving fluid flow problems using semi-supervised symbolic regression on sparse data — AIP Adv., 2019.
- Graph attention networks — ICLR, 2018.
- Inductive representation learning on large graphs — NIPS, 2017.
- Semi-supervised classification with graph convolutional networks — ICLR, 2017.
- Physics-AI symbiosis
5.2 Physics-informed ML and Scientific Computing
- A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks
- A Dual-Dimer method for training physics-constrained neural networks with minimax architecture
- A High-Efficient Hybrid Physics-Informed Neural Networks Based on Convolutional Neural Network
- A hybrid physics-informed neural network for nonlinear partial differential equation
- A Meta learning Approach for Physics-Informed Neural Networks (PINNs): Application to Parameterized PDEs
- A Novel Adaptive Causal Sampling Method for Physics-Informed Neural Networks
- A physics-aware learning architecture with input transfer networks for predictive modeling
- A physics-aware, probabilistic machine learning framework for coarse-graining high-dimensional systems in the Small Data regime
- A Physics-Data-Driven Bayesian Method for Heat Conduction Problems
- A rate of convergence of physics informed neural networks for the linear second order elliptic PDEs
- Accelerated Training of Physics-Informed Neural Networks (PINNS) using Meshless Discretizations
- Adversarial Multi-task Learning Enhanced Physics-informed Neural Networks for Solving Partial Differential Equations
- Adversarial uncertainty quantification in physics-informed neural networks
- B-PINNS: Bayesian physics-informed neural networks for forward and inverse PDE problems with noisy data
- Bayesian Physics Informed Neural Networks for real-world nonlinear dynamical systems
- Characterizing possible failure modes in physics-informed neural networks
- Convergence Rate of DeepONets for Learning Operators Arising from Advection-Diffusion Equations
- Deep neural network methods for solving forward and inverse problems of time fractional diffusion equations with conformable derivative
- Delta-PINNs: A new class of physics-informed neural networks for solving forward and inverse problems with noisy data
- DGM: A deep learning algorithm for solving partial differential equations
- Error analysis for physics informed neural networks (PINNs) approximating Kolmogorov PDEs
- Error Analysis of Deep Ritz Methods for Elliptic Equations
- Estimates on the generalization error of physics-informed neural networks for approximating a class of inverse problems for PDES
- Estimates on the generalization error of physics-informed neural networks for approximating PDEs
- Finite Basis Physics-Informed Neural Networks (FBPINNs): A scalable domain decomposition approach for solving differential equations
- Flow Field Tomography with Uncertainty Quantification using a Bayesian Physics-Informed Neural Network
- GPT-PINN: Generative Pre-Trained Physics-Informed Neural Networks toward non-intrusive Meta-learning of parametric PDEs
- Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems
- Hybrid physics-based and data-driven models for smart manufacturing: Modelling, simulation, and explainability
- HyperPINN: Learning parameterized differential equations with physics-informed hypernetworks
- Improved Deep Neural Networks with Domain Decomposition in Solving Partial Differential Equations
- INN: Interfaced neural networks as an accessible meshless approach for solving interface PDE problems
- Investigating and Mitigating Failure Modes in Physics-informed Neural Networks (PINNs)
- Is L2 Physics-Informed Loss Always Suitable for Training Physics-Informed Neural Network?
- Learning in Sinusoidal Spaces with Physics-Informed Neural Networks
- Learning Partial Differential Equations in Reproducing Kernel Hilbert Spaces
- LNN-PINN: A Unified Physics-Only Training Framework with Liquid Residual Blocks
- Meta-learning PINN loss functions
- META-PDE: LEARNING TO SOLVE PDES QUICKLY WITHOUT A MESH
- ModalPINN: An extension of physics-informed Neural Networks with enforced truncated Fourier decomposition for periodic flow reconstruction using a limited number of imperfect sensors
- Multi-Objective Loss Balancing for Physics-Informed Deep Learning
- Multi-output physics-informed neural networks for forward and inverse PDE problems with uncertainties
- Neural homogenization and the physics-informed neural network for the multiscale problems
- Novel Physics-Informed Artificial Neural Network Architectures for System and Input Identification of Structural Dynamics PDEs
- On Theory-training Neural Networks to Infer the Solution of Highly Coupled Differential Equations
- Parallel Physics-Informed Neural Networks via Domain Decomposition
- Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations
- Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations
- Physics-Augmented Learning: A New Paradigm Beyond Physics-Informed Learning
- Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modeling
- Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations
- Physics-informed graph neural Galerkin networks: A unified framework for solving PDE-governed forward and inverse problems
- Physics-informed machine learning
- Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications
- Physics-informed Neural Motion Planning on Constraint Manifolds
- Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
- Physics-Informed Neural Networks: A Review of Methodological Evolution, Theoretical Foundations, and Interdisciplinary Frontiers Toward Next-Generation Scientific Computing
- Physics-informed PointNet: A deep learning solver for steady-state incompressible flows and thermal fields on multiple sets of irregular geometries
- PID-GAN: A GAN Framework based on a Physics-informed Discriminator for Uncertainty Quantification with Physics
- PPINN: Parareal physics-informed neural network for time-dependent PDEs
- Robust Learning of Physics Informed Neural Networks
- Robust Regression with Highly Corrupted Data via Physics Informed Neural Networks
- RPINNS: Rectified-physics informed neural networks for solving stationary partial differential equations
- Self-adaptive loss balanced Physics-informed neural networks for the incompressible Navier-Stokes equations
- Separable Physics-Informed Neural Networks
- Simultaneous Neural Network Approximations in Sobolev Spaces
- Spectral PINNS: Fast Uncertainty Propagation with Physics-Informed Neural Networks
- SPINN: Sparse, Physics-based, and partially Interpretable Neural Networks for PDEs
- Stochastic Physics-Informed Neural Ordinary Differential Equations
- Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data
- Theory-guided hard constraint projection (HCP): A knowledge-based data-driven scientific machine learning method
- Training of physical neural networks
- Transfer learning based multi-fidelity physics informed deep neural network
- Transfer learning enhanced physics informed neural network for phase-field modeling of fracture
- Understanding and Mitigating Gradient Flow Pathologies in Physics-Informed Neural Networks
- Variational physics informed neural networks: The role of quadratures and test functions
- Wasserstein Generative Adversarial Uncertainty Quantification in Physics-Informed Neural Networks
- When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?
5.3 Domain-specific Applications and Future Directions
- A mixed formulation for physics-informed neural networks as a potential solver for engineering problems in heterogeneous domains: Comparison with finite element method
- A physically consistent framework for fatigue life prediction using probabilistic physics-informed neural network
- A Physics Informed Neural Network Approach to Solution and Identification of Biharmonic Equations of Elasticity
- A Physics-Informed Machine Learning Approach for Estimating Lithium-Ion Battery Temperature
- Advancing AI-Scientist Understanding: Multi-Agent LLMs with Interpretable Physics Reasoning
- Application of machine learning in solid state physics
- Calibrating Biophysical Models for Grape Phenology Prediction via Multi-Task Learning
- Coupled Lattice Boltzmann Modeling Framework for Pore-Scale Fluid Flow and Reactive Transport
- Creating a customisable freely-accessible Socratic AI physics tutor
- Deep learning for intrinsically disordered proteins: From improved predictions to deciphering conformational ensembles
- Deep Learning for Plasma Tomography and Disruption Prediction from Bolometer Data
- Discovering Phase Transitions with Unsupervised Learning
- Explicit physics-informed neural networks for nonlinear closure: The case of transport in tissues
- Foundation Neural-Networks Quantum States as a Unified Ansatz for Multiple Hamiltonians
- From Architectures to Applications: A Review of Neural Quantum States
- Graph Neural Networks in Particle Physics: Implementations, Innovations, and Challenges
- HeartUnloadNet: A Weakly-Supervised Cycle-Consistent Graph Network for Predicting Unloaded Cardiac Geometry from Diastolic States
- Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations
- Hierarchical symbolic regression for identifying key physical parameters correlated with bulk properties of perovskites
- Inverse modeling of nonisothermal multiphase poromechanics using physics-informed neural networks
- Investigating a New Approach to Quasinormal Modes: Physics-Informed Neural Networks
- KnowTD an actionable knowledge representation system for thermodynamics
- Lattice Thermal Conductivity Prediction using Symbolic Regression and Machine Learning
- Learning phase transitions by confusion
- Machine learning and the physical sciences
- Machine Learning Approaches to Predicting and Understanding the Mechanical Properties of Steels
- Machine learning phases of matter
- Machine learning-enhanced fully coupled fluid-solid interaction models for proppant dynamics in hydraulic fractures
- Mapping the design of electrolyte additive for stabilizing zinc anode in aqueous zinc ion batteries
- Neural-network quantum states for many-body physics
- Neuro - symbolic Al for materials modelling and processes design
- NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations
- PE-GPT: A Physics-Informed Interactive Large Language Model for Power Converter Modulation Design
- PFGM++: Unlocking the Potential of Physics-Inspired Generative Models
- Physically guided deep learning solver for time-dependent Fokker-Planck equation
- Physics-Informed Neural Network for Discovering Systems with Unmeasurable States with Application to Lithium-Ion Batteries
- Physics-informed neural network for lithium-ion battery degradation stable modeling and prognosis
- PHYSICS-INFORMED NEURAL NETWORK FOR NONLINEAR DYNAMICS IN FIBER OPTICS
- Physics-Informed Neural Networks for AC Optimal Power Flow
- Physics-informed neural networks for high-speed flows
- Physics-informed neural networks for the shallow-water equations on the sphere
- Poisson Flow Generative Models
- Prediction of porous media fluid flow using physics informed neural networks
- Recent advances and applications of machine learning in solid-state materials science
- Recent Advances in CO (2) Electroreduction Driven by Artificial Intelligence and Machine Learning
- Recent progress of artificial intelligence for liquid-vapor phase change heat transfer
- Review of Machine Learning for Real-Time Analysis at the Large Hadron Collider experiments ALICE, ATLAS, CMS and LHCb
- Searching for Exotic Particles in High-Energy Physics with Deep Learning
- Solving the Hubbard model with Neural Quantum States
- SpecCLIP: Aligning and Translating Spectroscopic Measurements for Stars
- State-of-the-art review on the use of AI-enhanced computational mechanics in geotechnical engineering
- Symbolic regression in materials science
- The AlphaPhysics Term Rewriting System for Marking Algebraic Expressions in Physics Exams
- Theory-training deep neural networks for an alloy solidification benchmark problem
- Towards neural Earth system modelling by integrating artificial intelligence in Earth system science
- Ultra-high-granularity detector simulation with intra-event aware generative adversarial network and self-supervised relational reasoning
- Using a Recurrent Neural Network to Reconstruct Quantum Dynamics of a Superconducting Qubit from Physical Observations
Contributors
Similar repos
Official implementation for the paper "Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI"
See the codeREADME
Table of Contents generated with DocToc
- Awesome AI+Physics
Awesome AI+Physics
A curated union of AI-for-physics literature, organized with the taxonomy from Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI.
1. Physical Perception
1.1 Object Recognition
- An expert ensemble for detecting anomalous scenes, interactions, and behaviors in autonomous driving — Int. J. Robotics Res., 2025.
- Qwen3-Omni technical report — arXiv:2509.17765, 2025.
- Towards visual discrimination and reasoning of real-world physical dynamics: Physics-grounded anomaly detection — CVPR, 2025.
- Cognition guided video anomaly detection framework for surveillance services — TSC, 2024.
- Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection — ECCV, 2024.
- Inter-object discriminative graph modeling for indoor scene recognition — KBS, 2024.
- View-invariant pixelwise anomaly detection in multi-object scenes with adaptive view synthesis — arXiv:2406.18012, 2024.
- BOP Challenge 2020 on 6D object localization — Eur. Conf. Comput. Vis. Workshops, 2020.
- Mask R-CNN — Proc. IEEE Int. Conf. Comput. Vis., 2017.
- Microsoft COCO: Common objects in context — ECCV, 2014.
- ImageNet: A large-scale hierarchical image database — CVPR, 2009.
1.2 Spatial Perception
- MMSI-Bench: A benchmark for multi-image spatial intelligence — ICLR, 2026.
- Contrastive learning of image representations guided by spatial relations — Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis., 2025.
- Multi-SpatialMLLM: multi-frame spatial understanding with multi-modal large language models — arXiv:2505.17015, 2025.
- Open3DVQA: A benchmark for comprehensive spatial reasoning with multimodal large language model in open space — arXiv:2503.11094, 2025.
- Spatial-MLLM: Boosting MLLM capabilities in visual-based spatial intelligence — arXiv:2505.23747, 2025.
- SpatialScore: Towards unified evaluation for multimodal spatial understanding — arXiv:2505.17012, 2025.
- ViewSpatial-Bench: Evaluating multi-perspective spatial localization in vision-language models — arXiv:2505.21500, 2025.
- Comprehensive multi-view representation learning — Inf. Fusion, 2023.
1.3 Identifying Intrinsic Property
- PhysID: Physics-based interactive dynamics from a single-view image — IEEE Int. Conf. Acoust., Speech, Signal Process., 2025.
- AI-based density recognition — WSCG, 2024.
- Intrinsic image diffusion for indoor single-view material estimation — CVPR, 2024.
- Practical measurements of translucent materials with inter-pixel translucency prior — CVPR, 2024.
- Vision-based approach for food weight estimation from 2D images — arXiv:2405.16478, 2024.
- Fabric surface characterization: assessment of deep learning-based texture representations using a challenging dataset — J. Text. Inst., 2021.
1.4 Dynamic Estimation
- PhysID: Physics-based interactive dynamics from a single-view image — IEEE Int. Conf. Acoust., Speech, Signal Process., 2025.
- Reasoning-enhanced object-centric learning for videos — Proc. ACM SIGKDD Conf. Knowl. Discov. Data Min., 2025.
- ContPhy: Continuum physical concept learning and reasoning from videos — ICML, 2024.
- I-PHYRE: Interactive physical reasoning — ICLR, 2024.
- Physion++: Evaluating physical scene understanding that requires online inference of different physical properties — NeurIPS, 2023.
- SlotFormer: Unsupervised visual dynamics simulation with object-centric models — ICLR, 2023.
- Physion: Evaluating physical prediction from vision in humans and machines — NeurIPS, 2021.
- A compositional object-based approach to learning physical dynamics — ICLR, 2017.
- Visual interaction networks: Learning a physics simulator from video — NIPS, 2017.
- Interaction networks for learning about objects, relations and physics — NIPS, 2016.
- Newtonian scene understanding: Unfolding the dynamics of objects in static images — CVPR, 2016.
2. Physics Reasoning
2.1 Symbolic Reasoning
- ABench-Physics: Benchmarking physical reasoning in LLMs via high-difficulty and dynamic physics problems — arXiv:2507.04766, 2025.
- LOCA-R: Near-Perfect Performance on the Chinese Physics Olympiad 2025 — arXiv:2511.10515, 2025.
- PHYBench: Holistic evaluation of physical perception and reasoning in large language models — arXiv:2504.16074, 2025.
- Physics Supernova: AI agent matches elite gold medalists at IPhO 2025 — arXiv:2509.01659, 2025.
- PhysicsEval: inference-time techniques to improve the reasoning proficiency of large language models on physics problems — Findings of IJCNLP, 2025.
- Symbolic or numerical? Understanding physics problem solving in reasoning LLMs — arXiv:2507.01334, 2025.
- UGPhysics: A comprehensive benchmark for undergraduate physics reasoning with large language models — arXiv:2502.00334, 2025.
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark — COLM, 2024.
- Knowledge graphs are all you need: Leveraging KGs in physics question answering — arXiv:2412.05453, 2024.
- LLMPhy: Complex physical reasoning using large language models and world models — arXiv:2411.08027, 2024.
- OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems — ACL, 2024.
- Steps are all you need: Rethinking STEM education with prompt engineering — arXiv:2412.05023, 2024.
- Could an artificial-intelligence agent pass an introductory physics course? — Phys. Rev. Phys. Educ. Res., 2023.
- Chain-of-thought prompting elicits reasoning in large language models — NeurIPS, 2022.
2.2 Multimodal-grounded Reasoning
- P1-VL: bridging visual perception and scientific reasoning in physics Olympiads — arXiv:2602.09443, 2026.
- TRACE: A framework for analyzing and enhancing stepwise reasoning in vision-language models — EACL, 2026.
- DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning — Nat., 2025.
- MMMU-Pro: A more robust multi-discipline multimodal understanding benchmark — ACL, 2025.
- Multimodal reasoning for science: Technical report and 1st place solution to the ICML 2025 SeePhys challenge — arXiv:2509.06079, 2025.
- PhysReason: A comprehensive benchmark towards physics-based reasoning — ACL, 2025.
- SeePhys: Does seeing help thinking?–Benchmarking vision-based physics reasoning — arXiv:2505.19099, 2025.
- MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI — CVPR, 2024.
- ComPhy: Compositional physical reasoning of objects and events from videos — ICLR, 2022.
- Grounding physical concepts of objects and events through dynamic visual reasoning — ICLR, 2021.
- CLEVRER: Collision events for video representation and reasoning — ICLR, 2020.
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
2.3 Causal and Counterfactual Reasoning
- CAUSAL3D: A comprehensive benchmark for causal learning from visual data — arXiv:2503.04852, 2025.
- PhySense: Principle-based physics reasoning benchmarking for large language models — arXiv:2505.24823, 2025.
- Using causal threads to explain changes in a dynamic system — Int. Conf. Asian Digit. Libr., 2023.
- CLEVRER-Humans: Describing physical and causal events the human way — NeurIPS, 2022.
- Dynamic visual reasoning by learning differentiable physics models from video and language — NeurIPS, 2021.
- Learning neural causal models from unknown interventions — arXiv:1910.01075, 2020.
- Causal modeling of dynamical systems — arXiv:1803.08784, 2018.
- GRASP: A novel benchmark for evaluating language GRounding And Situated Physics understanding in multimodal language models
- IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments
- PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding
2.4 Accelerate Physics Research
2.4.1 Hypothesis Generation
- ASP-assisted symbolic regression: Uncovering hidden physics in fluid mechanics — arXiv:2507.17777, 2025.
- Interpretable data-driven turbulence modeling for separated flows using symbolic regression with unit constraints — Acta Mech., 2025.
- Parsing the language of expression: Enhancing symbolic regression with domain-aware symbolic priors — arXiv:2503.09592, 2025.
- SRBench++: Principled benchmarking of symbolic regression with domain-expert interpretation — TEVC, 2025.
- A multi-agent framework for physical laws discovery — arXiv:2411.16416, 2024.
- Combining genetic algorithm and compressed sensing for features and operators selection in symbolic regression — arXiv:2403.15816, 2024.
- Contemporary symbolic regression methods and their relative performance — NeurIPS D B, 2021.
- AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity
- AI Feynman: A physics-inspired method for symbolic regression
- AI-Newton: A Concept-Driven Physical Law Discovery System without Prior Physical Knowledge
- Bayesian symbolic regression: Automated equation discovery from a physicists' perspective
- Discovering physical concepts with neural networks
- Intelligence, physics and information – the tradeoff between accuracy and simplicity in machine learning
- KAN 2.0: Kolmogorov-Arnold Networks Meet Science
- KAN: Kolmogorov-Arnold Networks
- Toward an AI Physicist for Unsupervised Learning
2.4.2 Streamline Experimentation
- Autonomous agents coordinating distributed discovery through emergent artifact exchange — arXiv:2603.14312, 2026.
- From agent-only social networks to autonomous scientific research: Lessons from OpenClaw and Moltbook, and the architecture of ClawdLab and Beach.Science — arXiv:2602.19810, 2026.
- Large language model driven development of turbulence models — Flow, 2025.
- Open source planning & control system with language agents for autonomous scientific discovery — ICML Workshop, 2025.
- ScienceAgentBench: Toward rigorous assessment of language agents for data-driven scientific discovery — ICLR, 2025.
- The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search — arXiv:2504.08066, 2025.
- AtomAgents: Alloy design and discovery through physics-aware multi-modal multi-agent artificial intelligence — arXiv:2407.10022, 2024.
- SciCode: A research coding benchmark curated by scientists — NeurIPS, 2024.
- The AI scientist: Towards fully automated open-ended scientific discovery — arXiv:2408.06292, 2024.
3. World Modeling
3.1 Image Generation
- Close the optical sensing domain gap by physics-grounded active stereo sensor simulation — TRO, 2023.
- ENVIDR: Implicit differentiable renderer with neural environment lighting — ICCV, 2023.
- Orbit: A unified simulation framework for interactive robot learning environments — RA-L, 2023.
- Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields — CVPR, 2022.
- CLIPScore: A reference-free evaluation metric for image captioning — EMNLP, 2021.
- Unity: A general platform for intelligent agents — arXiv:1809.02627, 2018.
- GANs trained by a two time-scale update rule converge to a local Nash equilibrium — NIPS, 2017.
- Indoor segmentation and support inference from RGBD images — ECCV, 2012.
- A reflectance model for computer graphics — TOG, 1982.
- Illumination for computer generated pictures — Commun. ACM, 1975.
- Generative AI for Validating Physics Laws
- PBR-NeRF: Inverse Rendering with Physics-Based Neural Fields
- Progressive Growing of GANs for Improved Quality, Stability, and Variation
3.2 Video Generation
- CogVideoX: text-to-video diffusion models with an expert transformer — ICLR, 2025.
- DINO-Foresight: Looking into the future with DINO — NeurIPS, 2025.
- Do generative video models learn physical principles from watching videos? — arXiv:2501.09038, 2025.
- LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models — IJCV, 2025.
- Morpheus: Benchmarking physical reasoning of video generative models with real physical experiments — arXiv:2504.02918, 2025.
- ProPhy: Progressive Physical Alignment for Dynamic World Simulation — arXiv:2512.05564, 2025.
- Think before you diffuse: LLMs-guided physics-aware video generation — arXiv:2505.21653, 2025.
- Towards world simulator: Crafting physical commonsense-based benchmark for video generation — ICML, 2025.
- VideoREPA: Learning physics for video generation through relational alignment with foundation models — NeurIPS, 2025.
- VLIPP: Towards physically plausible video generation with vision and language informed physical prior — ICCV, 2025.
- WorldModelBench: Judging video generation models as world models — arXiv:2502.20694, 2025.
- GPT4Motion: Scripting physical motions in text-to-video generation via Blender-oriented GPT planning — Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops, 2024.
- MotionCraft: Physics-Based zero-shot video generation — NeurIPS, 2024.
- Open-Sora: Democratizing efficient video production for all — arXiv:2412.20404, 2024.
- PhysGen: Rigid-Body physics-grounded image-to-video generation — ECCV, 2024.
- VideoPhy: Evaluating physical commonsense for video generation — arXiv:2406.03520, 2024.
- GAIA-1: A generative world model for autonomous driving — arXiv:2309.17080, 2023.
- VideoPoet: A large language model for zero-shot video generation — arXiv:2312.14125, 2023.
- IntrinsicAvatar: Physically Based Inverse Rendering of Dynamic Humans from Monocular Videos via Explicit Ray Tracing
- Self-Supervised Visual Planning with Temporal Skip Connections
- T2VPhysBench: A First-Principles Benchmark for Physical Consistency in Text-to-Video Generation
- The "Something Something" Video Database for Learning and Evaluating Visual Common Sense
- The Kinetics Human Action Video Dataset
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Unsupervised Learning of Video Representations using LSTMs
- VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation
3.3 Scene Reconstruction
- DriveDreamer4D: World models are effective data machines for 4D driving scene representation — CVPR, 2025.
- ReconDreamer: Crafting world models for driving scene reconstruction via online restoration — CVPR, 2025.
- PhyRecon: Physically plausible neural scene reconstruction — NeurIPS, 2024.
- Multiview neural surface reconstruction by disentangling geometry and appearance — NeurIPS, 2020.
- A multi-view stereo benchmark with high-resolution images and multi-camera videos — CVPR, 2017.
- GANs trained by a two time-scale update rule converge to a local Nash equilibrium — NIPS, 2017.
- Tanks and Temples: Benchmarking large-scale scene reconstruction — TOG, 2017.
- Large scale multi-view stereopsis evaluation — CVPR, 2014.
- MuJoCo: A physics engine for model-based control — IROS, 2012.
- A comparison and evaluation of multi-view stereo reconstruction algorithms — Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2006.
- The interpretation of structure from motion — Proc. R. Soc. Lond. B, Biol. Sci., 1979.
- PBR-NeRF: Inverse Rendering with Physics-Based Neural Fields
3.4 Physics-constrained Simulation
- PhyWorldBench: A comprehensive evaluation of physical realism in text-to-video models — ICLR, 2026.
- Back to the features: DINO as a foundation for video world models — arXiv:2507.19468, 2025.
- DEM-NeRF: A neuro-symbolic method for scientific discovery through physics-informed simulation — arXiv:2507.21350, 2025.
- FusionForce: End-to-end differentiable neural-symbolic layer for trajectory prediction — arXiv:2502.10156, 2025.
- How far is video generation from world model: A physical law perspective — ICML, 2025.
- Neural motion simulator: Pushing the limit of world models in reinforcement learning — CVPR, 2025.
- NeoPhysIx: An ultra fast 3D physical simulator as development tool for AI algorithms — arXiv:2411.05799, 2024.
- PhysORD: a neuro-symbolic approach for physics-infused motion prediction in off-road driving — IROS, 2024.
- Brax – A differentiable physics engine for large scale rigid body simulation — NeurIPS, 2021.
- LagNetViP: A Lagrangian neural network for video prediction — AAAI, 2020.
- Combining physical simulators and object-based networks for control — ICRA, 2019.
- PHYRE: A new benchmark for physical reasoning — NeurIPS, 2019.
- MuJoCo: A physics engine for model-based control — IROS, 2012.
- Brax: Massively Parallel Rigidbody Physics Simulation on Accelerator Hardware
- Building World Models with Neural Physics
- Isaac Sim: Robotics Simulation and Synthetic Data Generation
- MuJoCo: Advanced Physics Simulation
- PyBullet: Time Physics Simulation
4. Embodied Interaction
4.1 Robotics
- DriveVA: Video Action Models are Zero-Shot Drivers — arXiv:2604.04198, 2026.
- World Action Models are Zero-shot Policies — arXiv:2602.15922, 2026.
- World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetry — arXiv:2604.01985, 2026.
- Diffusion policy: Visuomotor policy learning via action diffusion — IJRR, 2025.
- EmbodiedBench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents — ICML, 2025.
- EMMOE: A comprehensive benchmark for embodied mobile manipulation in open environments — arXiv:2503.08604, 2025.
- OpenVLA: An open-source vision-language-action model — CoRL, 2025.
- Vision-language-action models: Concepts, progress, applications and challenges — arXiv:2505.04769, 2025.
- Active exploration for contact-rich tasks via online-adaptable models — ICRA, 2024.
- Open X-Embodiment: Robotic learning datasets and RT-X models — ICRA, 2024.
- π_0: A vision-language-action flow model for general robot control — arXiv:2410.24164, 2024.
- Active visuo-tactile interactive robotic perception for object property-aware manipulation — arXiv:2310.15551, 2023.
- Flow Matching for Generative Modeling — ICLR, 2023.
- ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills — International Conference on Learning Representations (ICLR), 2023.
- RT-1: Robotics Transformer for real-world control at scale — RSS, 2023.
- RT-2: Vision-language-action models transfer web knowledge to robotic control — CoRL, 2023.
- A generalist agent — Trans. Mach. Learn. Res., 2022.
- CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks — 2022.
- Do as I can, not as I say: Grounding language in robotic affordances — CoRL, 2022.
- Safety-critical kinematic control of robotic systems — IEEE Control Syst. Lett., 2022.
- BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments — 2021.
- BOP Challenge 2020 on 6D object localization — Eur. Conf. Comput. Vis. Workshops, 2020.
- GraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping — CVPR, 2020.
- RLBench: The Robot Learning Benchmark & Learning Environment — IEEE Robotics and Automation Letters, 2020.
- robosuite: A Modular Simulation Framework and Benchmark for Robot Learning — 2020.
- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning — 2019.
4.2 Navigation
- CL-CoTNav: Closed-loop hierarchical chain-of-thought for zero-shot object-goal navigation with vision-language models — arXiv:2504.09000, 2025.
- EmbodiedBench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents — ICML, 2025.
- FLAME: Learning to navigate with multimodal LLM in urban environments — AAAI, 2025.
- LGR: LLM-guided ranking of frontiers for object goal navigation — arXiv:2503.20241, 2025.
- NavBench: A unified robotics benchmark for reinforcement learning-based autonomous navigation — arXiv:2505.14526, 2025.
- NavCoT: Boosting LLM-based vision-and-language navigation via learning disentangled reasoning — TPAMI, 2025.
- NaVILA: Legged robot vision-language-action model for navigation — RSS, 2025.
- Research on navigation methods based on LLMs — arXiv:2504.15600, 2025.
- Stairway to success: zero-shot floor-aware object-goal navigation via LLM-driven coarse-to-fine exploration — arXiv:2505.23019, 2025.
- Uncovering the fragility of trustworthy LLMs through Chinese textual ambiguity — Agentic GenAI Evaluation Workshop KDD'25, 2025.
- X-MOBILITY: End-To-End Generalizable Navigation Via World Modeling — ICRA, 2025.
- Advancing object goal navigation through LLM-enhanced object affinities transfer — arXiv:2403.09971, 2024.
- Can LLM be a good path planner based on prompt engineering? Mitigating the hallucination for path planning — arXiv:2408.13184, 2024.
- HM3D-OVON: A dataset and benchmark for open-vocabulary object goal navigation — IROS, 2024.
- NavGPT: Explicit reasoning in vision-and-language navigation with large language models — AAAI, 2024.
- TWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfer — ICRA, 2024.
- UNMuTe: Unifying navigation and multimodal dialogue-like text generation — arXiv:2408.04423, 2024.
- VELMA: Verbalization embodiment of LLM agents for vision and language navigation in Street View — AAAI, 2024.
- Vision-and-language navigation today and tomorrow: A survey in the era of foundation models — arXiv:2407.07035, 2024.
- Object goal navigation with recursive implicit maps — IROS, 2023.
- ADAPT: Vision-language navigation with modality-aligned action prompts — CVPR, 2022.
- Chain-of-thought prompting elicits reasoning in large language models — NeurIPS, 2022.
- Goal-oriented vision-and-dialog navigation via reinforcement learning — Findings of EMNLP, 2022.
- Think global, act local: Dual-scale graph transformer for vision-and-language navigation — CVPR, 2022.
- History aware multimodal transformer for vision-and-language navigation — NeurIPS, 2021.
- The RobotSlang benchmark: Dialog-guided robot localization and navigation — CoRL, 2021.
- BabyWalk: Going farther in vision-and-language navigation by taking baby steps — Proc. 58th Annu. Meet. Assoc. Comput. Linguist., 2020.
- Beyond the nav-graph: Vision-and-language navigation in continuous environments — ECCV, 2020.
- Interactive Gibson benchmark: A benchmark for interactive navigation in cluttered environments — RA-L, 2020.
- Object goal navigation using goal-oriented semantic exploration — NeurIPS, 2020.
- REVERIE: Remote embodied visual referring expression in real indoor environments — CVPR, 2020.
- RoboTHOR: An open simulation-to-real embodied AI platform — CVPR, 2020.
- Room-Across-Room: Multilingual vision-and-language navigation with dense spatiotemporal grounding — Proc. 2020 Conf. Empir. Methods Nat. Lang. Process., 2020.
- Vision-and-dialog navigation — CoRL, 2020.
- Vision-dialog navigation by exploring cross-modal memory — CVPR, 2020.
- Learning to navigate unseen environments: Back translation with environmental dropout — Proc. Conf. North Amer. Chapter Assoc. Comput. Linguistics: Hum. Lang. Technol., 2019.
- TOUCHDOWN: Natural language navigation and spatial reasoning in visual street environments — CVPR, 2019.
- Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments — CVPR, 2018.
- AI2-THOR: An Interactive 3D Environment for Visual AI — arXiv:1712.05474, 2017.
- LLM ContextBridge: A Hybrid Approach for Intent and Dialogue Understanding in IVSR
- Matterport3D: Learning from RGB-D Data in Indoor Environments
- R2H: Building Multimodal Navigation Helpers that Respond to Help Requests
- Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation
4.3 Autonomous Driving
- AlphaDrive: Unleashing the power of VLMs in autonomous driving via reinforcement learning and reasoning — arXiv:2503.07608, 2025.
- AutoDrive-R^2: Incentivizing reasoning and self-reflection capacity for VLA model in autonomous driving — arXiv:2509.01944, 2025.
- Cosmos-Drive-Dreams: Scalable synthetic driving data generation with world foundation models — arXiv:2506.09042, 2025.
- DriveLMM-o1: A step-by-step reasoning dataset and large multimodal model for driving scenario understanding — arXiv:2503.10621, 2025.
- Driving in the occupancy world: Vision-centric 4D occupancy forecasting and planning via world models for autonomous driving — AAAI, 2025.
- DrivingGPT: Unifying driving world modeling and planning with multi-modal autoregressive transformers — ICCV, 2025.
- LeapVAD: A leap in autonomous driving via cognitive perception and dual-process thinking — TNNLS, 2025.
- Mastering diverse control tasks through world models — Nature, 2025.
- PRIMEDrive-CoT: A Precognitive Chain-of-Thought Framework for Uncertainty-Aware Object Interaction in Driving Scene Scenario — Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops, 2025.
- S2R-Bench: A sim-to-real evaluation benchmark for autonomous driving — Sci. Data, 2025.
- TeraSim-World: Worldwide safety-critical data synthesis for end-to-end autonomous driving — arXiv:2509.13164, 2025.
- Bench2Drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving — NeurIPS, 2024.
- DriveCoT: Integrating chain-of-thought reasoning with end-to-end driving — arXiv:2403.16996, 2024.
- DriveDreamer: Towards real-world-driven world models for autonomous driving — ECCV, 2024.
- OccLLaMA: An occupancy-language-action generative world model for autonomous driving — arXiv:2409.03272, 2024.
- Reason2Drive: Towards interpretable and chain-based reasoning for autonomous driving — ECCV, 2024.
- Think2Drive: Efficient reinforcement learning by thinking with latent world model for autonomous driving (in CARLA-v2) — ECCV, 2024.
- A survey on safety-critical driving scenario generation—A methodological perspective — TITS, 2023.
- Interaction-aware trajectory planning for autonomous vehicles with analytic integration of neural networks into model predictive control — ICRA, 2023.
- Deep reinforcement learning for autonomous driving: A survey — TITS, 2022.
- MotionSC: data set and network for real-time semantic mapping in dynamic environments — RA-L, 2022.
- Wayformer: Motion forecasting via simple & efficient attention networks — arXiv:2207.05844, 2022.
- NuPlan: a closed-loop ML-based planning benchmark for autonomous vehicles — arXiv:2106.11810, 2021.
- A survey of deep learning techniques for autonomous driving — J. Field Robot., 2020.
- Human motion trajectory prediction: a survey — IJRR, 2020.
- Learning lane graph representations for motion forecasting — ECCV, 2020.
- nuScenes: A multimodal dataset for autonomous driving — CVPR, 2020.
- Scalability in perception for autonomous driving: Waymo open dataset — CVPR, 2020.
- ChauffeurNet: Learning to drive by imitating the best and synthesizing the worst — RSS, 2019.
- A framework for automated driving system testable cases and scenarios — 2018.
- AirSim: High-fidelity visual and physical simulation for autonomous vehicles — Field Serv. Robot., 2018.
- World models — arXiv:1803.10122, 2018.
- CARLA: An open urban driving simulator — CoRL, 2017.
- CommonRoad: Composable benchmarks for motion planning on roads — IEEE Intell. Veh. Symp., 2017.
- A survey of motion planning and control techniques for self-driving urban vehicles — IEEE Trans. Intell. Veh., 2016.
- End to end learning for self-driving cars — arXiv:1604.07316, 2016.
- Maximum entropy inverse reinforcement learning — AAAI, 2008.
- A survey of industrial model predictive control technology — Control Eng. Pract., 2003.
- Congested traffic states in empirical observations and microscopic simulations — Phys. Rev. E, 2000.
- Rapidly-exploring random trees: A new tool for path planning — Research Report 9811, 1998.
- Real-time obstacle avoidance for manipulators and mobile robots — IJRR, 1986.
- Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting
- Argoverse: 3D Tracking and Forecasting with Rich Maps
- BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning
- BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layout
- Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion
- DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Model
- DreamDrive: Generative 4D Scene Modeling from Street View Images
- DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model
- DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT
- HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation
- HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving
- Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving
- OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving
- One Thousand and One Hours: Self-driving Motion Prediction Dataset
- RenderWorld: World Model with Self-Supervised 3D Label
- RoboTron-Sim: Improving Real-World Driving via Simulated Hard-Case
- Stag-1: Towards Realistic 4D Driving Simulation with Video Generation Model
- The ApolloScape Open Dataset for Autonomous Driving and its Application
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)
- UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generation
- Vision meets robotics: The KITTI dataset
- Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability
5. Survey-wide Foundations and Repository Extensions
5.1 Survey-wide Foundations and Cross-cutting References
- From System 1 to System 2: A survey of reasoning large language models — TPAMI, 2026.
- Object detection with multimodal large vision-language models: An in-depth review — Inf. Fusion, 2026.
- The artificial intelligence cognitive examination: A survey on the evolution of multimodal evaluation from recognition to reasoning — IEEE Access, 2026.
- 3D and 4D world modeling: A survey — arXiv:2509.07996, 2025.
- A physics-guided multimodal transformer path to weather and climate sciences — arXiv:2504.14174, 2025.
- A survey of embodied learning for object-centric robotic manipulation — Mach. Intell. Res., 2025.
- A survey on world models grounded in acoustic physical information — arXiv:2506.13833, 2025.
- A survey: Learning embodied intelligence from physical simulators and world models — arXiv:2507.00917, 2025.
- Embodied AI: From LLMs to world models — IEEE Circuits Syst. Mag., 2025.
- Foundation model driven robotics: A comprehensive review — arXiv:2507.10087, 2025.
- Foundation models for autonomous driving perception: A survey through core capabilities — arXiv:2509.08302, 2025.
- From 2D to 3D cognition: A brief survey of general world models — arXiv:2506.20134, 2025.
- From perception to cognition: A survey of vision-language interactive reasoning in multimodal large language models — arXiv:2509.25373, 2025.
- Generative physical AI in vision: A survey — arXiv:2501.10928, 2025.
- Large model empowered embodied AI: A survey on decision-making and embodied learning — arXiv:2508.10399, 2025.
- Large physics models: towards a collaborative approach with large language models and foundation models — EPJ C, 2025.
- MINT-CoT: Enabling interleaved visual tokens in mathematical chain-of-thought reasoning — arXiv:2506.05331, 2025.
- PhyBlock: A progressive benchmark for physical understanding and planning via 3D block assembly — NeurIPS, 2025.
- Probing perceptual constancy in large vision language models — arXiv:2502.10273, 2025.
- Qwen2.5-VL technical report — arXiv:2502.13923, 2025.
- Reinforcement learning meets large language models: A survey of advancements and applications across the LLM lifecycle — arXiv:2509.16679, 2025.
- Toward embodied AGI: A review of embodied AI and the road ahead — arXiv:2505.14235, 2025.
- Understanding world or predicting future? A comprehensive survey of world models — CSUR, 2025.
- A survey on large language model based autonomous agents — Front. Comput. Sci., 2024.
- A survey on multimodal large language models — NSR, 2024.
- A survey on robotics with foundation models: toward embodied AI — arXiv:2402.02385, 2024.
- AI meets physics: a comprehensive survey — Artif. Intell. Rev., 2024.
- Aligning cyber space with physical world: A comprehensive survey on embodied AI — arXiv:2407.06886, 2024.
- From efficient multimodal models to world models: A survey — arXiv:2407.00118, 2024.
- GPT-4o system card — arXiv:2410.21276, 2024.
- Is Sora a world simulator? A comprehensive survey on general world models and beyond — arXiv:2405.03520, 2024.
- OpenAI o1 system card — arXiv:2412.16720, 2024.
- PhyGrasp: generalizing robotic grasping with physics-informed large multimodal models — arXiv:2402.16836, 2024.
- ShapeLLM: Universal 3D object understanding for embodied interaction — ECCV, 2024.
- A survey on deep reinforcement learning algorithms for robotic manipulation — Sensors, 2023.
- Towards reasoning in large language models: A survey — Findings of ACL, 2023.
- Visual instruction tuning — NeurIPS, 2023.
- A survey of embodied AI: From simulators to research tasks — TETCI, 2022.
- A survey on machine learning approaches for modelling intuitive physics — IJCAI, 2022.
- Functional optimization of fluidic devices with differentiable Stokes flow — TOG, 2020.
- Scalable differentiable physics for learning and control — ICML, 2020.
- Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data — CMAME, 2020.
- Machine learning for data-driven discovery in solid Earth geoscience — Science, 2019.
- Solving fluid flow problems using semi-supervised symbolic regression on sparse data — AIP Adv., 2019.
- Graph attention networks — ICLR, 2018.
- Inductive representation learning on large graphs — NIPS, 2017.
- Semi-supervised classification with graph convolutional networks — ICLR, 2017.
- Physics-AI symbiosis
5.2 Physics-informed ML and Scientific Computing
- A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks
- A Dual-Dimer method for training physics-constrained neural networks with minimax architecture
- A High-Efficient Hybrid Physics-Informed Neural Networks Based on Convolutional Neural Network
- A hybrid physics-informed neural network for nonlinear partial differential equation
- A Meta learning Approach for Physics-Informed Neural Networks (PINNs): Application to Parameterized PDEs
- A Novel Adaptive Causal Sampling Method for Physics-Informed Neural Networks
- A physics-aware learning architecture with input transfer networks for predictive modeling
- A physics-aware, probabilistic machine learning framework for coarse-graining high-dimensional systems in the Small Data regime
- A Physics-Data-Driven Bayesian Method for Heat Conduction Problems
- A rate of convergence of physics informed neural networks for the linear second order elliptic PDEs
- Accelerated Training of Physics-Informed Neural Networks (PINNS) using Meshless Discretizations
- Adversarial Multi-task Learning Enhanced Physics-informed Neural Networks for Solving Partial Differential Equations
- Adversarial uncertainty quantification in physics-informed neural networks
- B-PINNS: Bayesian physics-informed neural networks for forward and inverse PDE problems with noisy data
- Bayesian Physics Informed Neural Networks for real-world nonlinear dynamical systems
- Characterizing possible failure modes in physics-informed neural networks
- Convergence Rate of DeepONets for Learning Operators Arising from Advection-Diffusion Equations
- Deep neural network methods for solving forward and inverse problems of time fractional diffusion equations with conformable derivative
- Delta-PINNs: A new class of physics-informed neural networks for solving forward and inverse problems with noisy data
- DGM: A deep learning algorithm for solving partial differential equations
- Error analysis for physics informed neural networks (PINNs) approximating Kolmogorov PDEs
- Error Analysis of Deep Ritz Methods for Elliptic Equations
- Estimates on the generalization error of physics-informed neural networks for approximating a class of inverse problems for PDES
- Estimates on the generalization error of physics-informed neural networks for approximating PDEs
- Finite Basis Physics-Informed Neural Networks (FBPINNs): A scalable domain decomposition approach for solving differential equations
- Flow Field Tomography with Uncertainty Quantification using a Bayesian Physics-Informed Neural Network
- GPT-PINN: Generative Pre-Trained Physics-Informed Neural Networks toward non-intrusive Meta-learning of parametric PDEs
- Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems
- Hybrid physics-based and data-driven models for smart manufacturing: Modelling, simulation, and explainability
- HyperPINN: Learning parameterized differential equations with physics-informed hypernetworks
- Improved Deep Neural Networks with Domain Decomposition in Solving Partial Differential Equations
- INN: Interfaced neural networks as an accessible meshless approach for solving interface PDE problems
- Investigating and Mitigating Failure Modes in Physics-informed Neural Networks (PINNs)
- Is L2 Physics-Informed Loss Always Suitable for Training Physics-Informed Neural Network?
- Learning in Sinusoidal Spaces with Physics-Informed Neural Networks
- Learning Partial Differential Equations in Reproducing Kernel Hilbert Spaces
- LNN-PINN: A Unified Physics-Only Training Framework with Liquid Residual Blocks
- Meta-learning PINN loss functions
- META-PDE: LEARNING TO SOLVE PDES QUICKLY WITHOUT A MESH
- ModalPINN: An extension of physics-informed Neural Networks with enforced truncated Fourier decomposition for periodic flow reconstruction using a limited number of imperfect sensors
- Multi-Objective Loss Balancing for Physics-Informed Deep Learning
- Multi-output physics-informed neural networks for forward and inverse PDE problems with uncertainties
- Neural homogenization and the physics-informed neural network for the multiscale problems
- Novel Physics-Informed Artificial Neural Network Architectures for System and Input Identification of Structural Dynamics PDEs
- On Theory-training Neural Networks to Infer the Solution of Highly Coupled Differential Equations
- Parallel Physics-Informed Neural Networks via Domain Decomposition
- Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations
- Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations
- Physics-Augmented Learning: A New Paradigm Beyond Physics-Informed Learning
- Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modeling
- Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations
- Physics-informed graph neural Galerkin networks: A unified framework for solving PDE-governed forward and inverse problems
- Physics-informed machine learning
- Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications
- Physics-informed Neural Motion Planning on Constraint Manifolds
- Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
- Physics-Informed Neural Networks: A Review of Methodological Evolution, Theoretical Foundations, and Interdisciplinary Frontiers Toward Next-Generation Scientific Computing
- Physics-informed PointNet: A deep learning solver for steady-state incompressible flows and thermal fields on multiple sets of irregular geometries
- PID-GAN: A GAN Framework based on a Physics-informed Discriminator for Uncertainty Quantification with Physics
- PPINN: Parareal physics-informed neural network for time-dependent PDEs
- Robust Learning of Physics Informed Neural Networks
- Robust Regression with Highly Corrupted Data via Physics Informed Neural Networks
- RPINNS: Rectified-physics informed neural networks for solving stationary partial differential equations
- Self-adaptive loss balanced Physics-informed neural networks for the incompressible Navier-Stokes equations
- Separable Physics-Informed Neural Networks
- Simultaneous Neural Network Approximations in Sobolev Spaces
- Spectral PINNS: Fast Uncertainty Propagation with Physics-Informed Neural Networks
- SPINN: Sparse, Physics-based, and partially Interpretable Neural Networks for PDEs
- Stochastic Physics-Informed Neural Ordinary Differential Equations
- Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data
- Theory-guided hard constraint projection (HCP): A knowledge-based data-driven scientific machine learning method
- Training of physical neural networks
- Transfer learning based multi-fidelity physics informed deep neural network
- Transfer learning enhanced physics informed neural network for phase-field modeling of fracture
- Understanding and Mitigating Gradient Flow Pathologies in Physics-Informed Neural Networks
- Variational physics informed neural networks: The role of quadratures and test functions
- Wasserstein Generative Adversarial Uncertainty Quantification in Physics-Informed Neural Networks
- When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?
5.3 Domain-specific Applications and Future Directions
- A mixed formulation for physics-informed neural networks as a potential solver for engineering problems in heterogeneous domains: Comparison with finite element method
- A physically consistent framework for fatigue life prediction using probabilistic physics-informed neural network
- A Physics Informed Neural Network Approach to Solution and Identification of Biharmonic Equations of Elasticity
- A Physics-Informed Machine Learning Approach for Estimating Lithium-Ion Battery Temperature
- Advancing AI-Scientist Understanding: Multi-Agent LLMs with Interpretable Physics Reasoning
- Application of machine learning in solid state physics
- Calibrating Biophysical Models for Grape Phenology Prediction via Multi-Task Learning
- Coupled Lattice Boltzmann Modeling Framework for Pore-Scale Fluid Flow and Reactive Transport
- Creating a customisable freely-accessible Socratic AI physics tutor
- Deep learning for intrinsically disordered proteins: From improved predictions to deciphering conformational ensembles
- Deep Learning for Plasma Tomography and Disruption Prediction from Bolometer Data
- Discovering Phase Transitions with Unsupervised Learning
- Explicit physics-informed neural networks for nonlinear closure: The case of transport in tissues
- Foundation Neural-Networks Quantum States as a Unified Ansatz for Multiple Hamiltonians
- From Architectures to Applications: A Review of Neural Quantum States
- Graph Neural Networks in Particle Physics: Implementations, Innovations, and Challenges
- HeartUnloadNet: A Weakly-Supervised Cycle-Consistent Graph Network for Predicting Unloaded Cardiac Geometry from Diastolic States
- Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations
- Hierarchical symbolic regression for identifying key physical parameters correlated with bulk properties of perovskites
- Inverse modeling of nonisothermal multiphase poromechanics using physics-informed neural networks
- Investigating a New Approach to Quasinormal Modes: Physics-Informed Neural Networks
- KnowTD an actionable knowledge representation system for thermodynamics
- Lattice Thermal Conductivity Prediction using Symbolic Regression and Machine Learning
- Learning phase transitions by confusion
- Machine learning and the physical sciences
- Machine Learning Approaches to Predicting and Understanding the Mechanical Properties of Steels
- Machine learning phases of matter
- Machine learning-enhanced fully coupled fluid-solid interaction models for proppant dynamics in hydraulic fractures
- Mapping the design of electrolyte additive for stabilizing zinc anode in aqueous zinc ion batteries
- Neural-network quantum states for many-body physics
- Neuro - symbolic Al for materials modelling and processes design
- NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations
- PE-GPT: A Physics-Informed Interactive Large Language Model for Power Converter Modulation Design
- PFGM++: Unlocking the Potential of Physics-Inspired Generative Models
- Physically guided deep learning solver for time-dependent Fokker-Planck equation
- Physics-Informed Neural Network for Discovering Systems with Unmeasurable States with Application to Lithium-Ion Batteries
- Physics-informed neural network for lithium-ion battery degradation stable modeling and prognosis
- PHYSICS-INFORMED NEURAL NETWORK FOR NONLINEAR DYNAMICS IN FIBER OPTICS
- Physics-Informed Neural Networks for AC Optimal Power Flow
- Physics-informed neural networks for high-speed flows
- Physics-informed neural networks for the shallow-water equations on the sphere
- Poisson Flow Generative Models
- Prediction of porous media fluid flow using physics informed neural networks
- Recent advances and applications of machine learning in solid-state materials science
- Recent Advances in CO (2) Electroreduction Driven by Artificial Intelligence and Machine Learning
- Recent progress of artificial intelligence for liquid-vapor phase change heat transfer
- Review of Machine Learning for Real-Time Analysis at the Large Hadron Collider experiments ALICE, ATLAS, CMS and LHCb
- Searching for Exotic Particles in High-Energy Physics with Deep Learning
- Solving the Hubbard model with Neural Quantum States
- SpecCLIP: Aligning and Translating Spectroscopic Measurements for Stars
- State-of-the-art review on the use of AI-enhanced computational mechanics in geotechnical engineering
- Symbolic regression in materials science
- The AlphaPhysics Term Rewriting System for Marking Algebraic Expressions in Physics Exams
- Theory-training deep neural networks for an alloy solidification benchmark problem
- Towards neural Earth system modelling by integrating artificial intelligence in Earth system science
- Ultra-high-granularity detector simulation with intra-event aware generative adversarial network and self-supervised relational reasoning
- Using a Recurrent Neural Network to Reconstruct Quantum Dynamics of a Superconducting Qubit from Physical Observations