Awesome-Video-Robotic-Papers

This repository compiles a list of seminal and cutting-edge papers that explore the application of video technology in the field of robotics. Continual improvements are being made to this repository, and contributions are welcome. If you come across any relevant papers that should be included, please don't hesitate to open an issue.

Table of Contents

1. Review Papers
2. Robot Arm
3. SPOT
4. Dataset
5. Other Useful Sources

Review Papers

• Towards Generalist Robot Learning from Internet Video: A Survey

  • Robert McCarthy, Daniel C.H. Tan, Dominik Schmidt, Fernando Acero, Nathan Herr, Yilun Du, Thomas G. Thuruthel, Zhibin Li
  • Paper

• Learning by Watching: A Review of Video-based Learning Approaches for Robot Manipulation

  • Chrisantus Eze, Christopher Crick
  • Paper

Robot Arm

• Let Me Help You! Neuro-Symbolic Short-Context Action Anticipation

  • Sarthak Bhagat, Samuel Li, Joseph Campbell, Yaqi Xie, Katia Sycara, Simon Stepputtis
  • Paper
  • Website
  • Code
  • IEEE Robotics and Automation Letters
  • The Robotics Institute, Carnegie Melon University

• Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Tasks

  • Kuan Fang, Patrick Yin, Ashvin Nair, Homer Walke, Gengchen Yan, Sergey Levine
  • Paper
  • Code
  • CoRL 2022
  • UC Berkeley

• VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training

  • Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, Amy Zhang
  • Paper
  • Website
  • Code
  • ICLR 2023, Notable-Top-25% (Spotlight)
  • FAIR, Meta AI || University of Pennsylvania

• SOAR: Autonomous Improvement of Instruction Following Skills via Foundation Models

  • Zhiyuan Zhou, Pranav Atreya, Abraham Lee, Homer Walke, Oier Mees, Sergey Levine
  • Paper
  • Website
  • Code
  • Dataset
  • UC Berkeley

• HRP: Human Affordances for Robotic Pre-Training

  • Mohan Kumar Srirama, Sudeep Dasari, Shikhar Bahl, Abhinav Gupta
  • Paper
  • Robotics Science and Systems 2024
  • CMU

• Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos

  • Changan Chen, Puyuan Peng, Ami Baid, Zihui Xue, Wei-Ning Hsu, David Harwath, Kristen Grauman
  • Paper
  • Website
  • [Code] not release yet
  • University of Texas at Austin || FAIR, Meta AI

• This&That: Language-Gesture Controlled Video Generation for Robot Planning

  • Boyang Wang, Nikhil Sridhar, Chao Feng, Mark Van der Merwe, Adam Fishman, Nima Fazeli, Jeong Joon Park
  • Paper
  • Website
  • [Code] not release yet
  • University of Michigan || University of Washington

• Policy Composition From and For Heterogeneous Robot Learning

  • Lirui Wang, Alan Zhao, Yilun Du, Ted Adelson, Russ Tedrake
  • Paper
  • Website
  • MIT CSAIL
  • Robotics: Science and Systems (R:SS), 2024

• Simultaneous Localization and Affordance Prediction for Tasks in Egocentric Video

  • Zachary Chavis, Hyun Soo Park, and Stephen J. Guy
  • Paper
  • Department of Computer Science and Engineering, University of Minnesota

• Flow as the Cross-domain Manipulation Interface

  • Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gordon Wetzstein, Manuela Veloso, Shuran Song
  • Paper
  • Website
  • Stanford University || Columbia University || JP Morgan AI Research || Carnegie Mellon University

• R+X: Retrieval and Execution from Everyday Human Videos

  • Georgios Papagiannis, Norman Di Palo, Pietro Vitiello, Edward Johns
  • Paper
  • Website
  • the Robot Learning Lab at Imperial College London

• Octo: An Open-Source Generalist Robot Policy

  • Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, Sergey Levine
  • Paper
  • Website
  • Code
  • UC Berkeley || Stanford || Carnegie Mellon University || Google Deepmind

• HRP: Human Affordances for Robotic Pre-Training

  • Mohan Kumar Srirama, Sudeep Dasari, Shikhar Bahl, Abhinav Gupta
  • Paper
  • Website
  • Code
  • Carnegie Mellon University
  • RSS 2024

• RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulation

  • Yuxuan Kuang*, Junjie Ye*, Haoran Geng*, Jiageng Mao, Congyue Deng, Leonidas Guibas, He Wang, Yue Wang
  • Paper
  • Website
  • Code
  • University of Southern California || Peking University || Stanford University

• Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers

  • Vidhi Jain, Maria Attarian, Nikhil J Joshi, Ayzaan Wahid, Danny Driess, Quan Vuong, Pannag R Sanketi, Pierre Sermanet, Stefan Welker, Christine Chan, Igor Gilitschenski, Yonatan Bisk, Debidatta Dwibedi
  • Paper
  • Website
  • Google DeepMind || Carnegie Mellon University || University of Toronto

• OpenVLA: An Open-Source Vision-Language-Action Model

  • Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, Chelsea Finn
  • Paper
  • Website
  • Code
  • Stanford University || UC Berkeley || Toyota Research Institute || Google DeepMind || Physical Intelligence || MIT

• Video Language Planning

  • Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B. Tenenbaum, Leslie Kaelbling, Andy Zeng, Jonathan Tompson
  • Paper
  • Website
  • Code
  • Google Deepmind || MIT || UC Berkeley

• Manipulate-Anything: Automating Real-World Robots using Vision-Language Models

  • Jiafei Duan, Wentao Yuan, Wilbert Pumacay, Yi Ru Wang, Kiana Ehsani, Dieter Fox, Ranjay Krishna
  • Paper
  • Website
  • University of Washington || NVIDIA || Allen Institute for Artifical Intelligence || Universidad Católica San Pablo

• Dreamitate: Real-World Visuomotor Policy Learning via Video Generation

  • Junbang Liang, Ruoshi Liu, Ege Ozguroglu, Sruthi Sudhakar, Achal Dave, Pavel Tokmakov, Shuran Song, Carl Vondrick
  • Paper
  • Website
  • Code
  • Columbia University || Toyota Research Institute || Stanford University,

• Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation

  • Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, Tao Kong
  • Paper
  • Website
  • Code
  • ByteDance Research

• Large-Scale Actionless Video Pre-Training via Discrete Diffusion for Efficient Policy Learning

  • Haoran He, Chenjia Bai, Ling Pan, Weinan Zhang, Bin Zhao, Xuelong Li
  • Paper
  • Website
  • Hong Kong University of Science and Technology || Shanghai Artificial Intelligence Laboratory || Shanghai Jiao Tong University || Northwestern Polytechnical University || Institute of Artificial In- telligence (TeleAI), China Telecom Corp Ltd.

• ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data

  • Zeyi Liu, Cheng Chi, Eric Cousineau, Naveen Kuppuswamy, Benjamin Burchfiel, Shuran Song
  • Paper
  • Website
  • Code
  • Dataset
  • Stanford University || Columbia University || Toyota Research Institute

• Vision-based Manipulation from Single Human Video with Open-World Object Graphs

  • Yifeng Zhu, Arisrei Lim, Peter Stone, Yuke Zhu
  • Paper
  • Website
  • The University of Texas at Austin || Sony AI

• Learning to Act from Actionless Videos through Dense Correspondences

  • Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, Joshua B. Tenenbaum
  • Paper
  • Website
  • Code
  • National Taiwan University | MIT

SPOT

• Track2Act: Predicting Point Tracks from Internet Videos Enables Diverse Zero-shot Manipulation
  • Homanga Bharadhwaj, Roozbeh Mottaghi*, Abhinav Gupta*, Shubham Tulsiani*
  • Paper
  • Website
  • Code

Dataset

• The Ingredients for Robotic Diffusion Transformers

  • JSudeep Dasari, Oier Mees, Sebastian Zhao, Mohan Kumar Srirama, Sergey Levine
  • Paper
  • Website
  • Code
  • 

• Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding

  • Joshua Jones*, Oier Mees*, Carmelo Sferrazza*, Kyle Stachowicz, Pieter Abbeel, Sergey Levine
  • Paper
  • Website
  • Code
  • 

• DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

  • Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O'Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martín-Martín, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, Chelsea Finn
  • Paper
  • Website
  • check the website and get much sources
  • 

• Open X-Embodiment: Robotic Learning Datasets and RT-X Models

  • Open X-Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang, Guanzhi Wang, Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homanga Bharadhwaj, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jay Vakil et al. (192 additional authors not shown)
  • Paper
  • Website
  • Code
  • 

• BridgeData V2: A Dataset for Robot Learning at Scale

  • Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, Sergey Levine
  • Paper
  • Website
  • Code
  • 

• Ego4DSounds:A diverse egocentric dataset with high action-audio correspondence

  • Changan Chen, Puyuan Peng, Ami Baid, Zihui Xue, Wei-Ning Hsu, David Harwath, Kristen Grauman
  • Paper
  • Website
  • Code
  • 

• RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot

  • Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Chenxi Wang, Junbo Wang, Haoyi Zhu, Cewu Lu
  • Paper
  • Website
  • API
  • Dataset
  • Shanghai Jiao Tong University
  • 

Other Useful Sources

• Awesome-VideoLLM-Papers
• Awesome-LLMs-for-Video-Understanding
• VLM-Eval: A General Evaluation on Video Large Language Models
• LLMs Meet Multimodal Generation and Editing: A Survey