A curated, taxonomy-driven collection of language-conditioned robot manipulation papers, code, simulators, and benchmarks – tracking the literature behind
“Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation” (arXiv:2312.10807).
If you find this repo useful, please:
so more people can discover and build on this survey!
This repository is built around the survey:
Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation
Hongkuan Zhou, Xiangtong Yao, Oier Mees, Yuan Meng, Ted Xiao, Yonatan Bisk, Jean Oh, Edward Johns, Mohit Shridhar, Dhruv Shah, Jesse Thomason, Kai Huang, Joyce Chai, Zhenshan Bing, Alois Knoll
The structure of the survey organizes methods by the role language plays in the system.
At a high level, language can:
Evaluate what the robot is doing
-> Language for state evaluation
Language becomes a reward, cost, or scoring function, used to measure task progress, preferences, or goal satisfaction.
Specify how the robot should act
-> Language as a policy condition (Sec. 5)
Language is fed directly into the policy, shaping the action distribution at each step (e.g., language-conditioned RL, BC, diffusion policies).
Help the robot think and plan
-> Language for cognitive planning and reasoning (Sec. 6)
Language is used as an internal reasoning medium: planning, decomposition, querying knowledge bases, or manipulating symbolic structures.
Below, we briefly summarize each role and show how it maps to the sections in this repo.
Hierarchical understanding in robotic manipulation: A knowledge-based framework [paper]
Semantic Grasping Via a Knowledge Graph of Robotic Manipulation: A Graph Representation Learning Approach [paper]
Knowledge Acquisition and Completion for Long-Term Human-Robot Interactions using Knowledge Graph Embedding [paper]
Tell me dave: Context-sensitive grounding of natural language to manipulation instructions [paper]
Neuro-symbolic procedural planning with commonsense prompting [paper]
Reinforcement Learning Based Navigation with Semantic Knowledge of Indoor Environments [paper]
Learning Neuro-Symbolic Skills for Bilevel Planning [[paper]](Learning Neuro-Symbolic Skills for Bilevel Planning)
Learning Neuro-symbolic Programs for Language Guided Robot Manipulation [paper] [code]
Long-term robot manipulation task planning with scene graph and semantic knowledge [paper]
Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition [paper] [code]
ProgPrompt: program generation for situated robot task planning using large language models [paper]
Data-Agnostic Robotic Long-Horizon Manipulation with Vision-Language-Guided Closed-Loop Feedback [paper] [code]
Growing with Your Embodied Agent: A Human-in-the-Loop Lifelong Code Generation Framework for Long-Horizon Manipulation Skills [paper] [code]
| Optimization (Direction) | Article | Time | Observation | Action Generation | CoT | FP | MEM | MD | Pretraining CE | Scenarios MS | Scenarios RW | Execution CE |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Data Source Augmentation | EgoVLA | 2025-07 | RGB, ROB, TX | DP | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
| H-RDT | 2025-08 | RGB, ROB, TX | FM | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| Shortcut | 2025-08 | RGB, TX | - | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | |
| Spatial Understanding | SpatialVLA | 2025-01 | RGB, ROB, TX | AR | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ |
| PointVLA | 2025-05 | RGBD, ROB, TX | DM | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| BridgeVLA | 2025-06 | RGB, ROB, TX | DP | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | |
| GeoVLA | 2025-08 | RGBD, ROB, TX | DM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | |
| Multimodal Sensing & Fusion | VTLA | 2025-05 | RGB, TX | AR | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ |
| Tactile-VLA | 2025-07 | RGB, ROB, TX | FM | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | |
| OmniVTLA | 2025-08 | RGB, ROB, TX | FM | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ | |
| ForceVLA | 2025-09 | RGBD, ROB, TX | FM | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | |
| FuSe VLA | 2025-01 | RGBD, ROB, TX | AR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | |
| Long-horizon task solving | LoHoVLA | 2025-05 | RGB, ROB, TX | AR | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ |
| Long-VLA | 2025-08 | RGB, TX | DM | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | |
| DexVLA | 2025-08 | RGB, ROB, TX | DM | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| MemoryVLA | 2025-08 | RGB, TX | DM | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| DiffusionVLA | 2024-12 | RGBD, TX | DM + AR | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | |
| Knowledge Preserving | ChatVLA | 2025-02 | RGB, TX | DM | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ |
| ChatVLA-2 | 2025-05 | RGB, TX | DM | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | |
| Insulating | 2025-05 | RGB, ROB, TX | FM + AR | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| InstructVLA | 2025-07 | RGB, ROB, TX | FM | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | |
| GR-3 | 2025-07 | RGB, ROB, TX | FM | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | |
| Reasoning & World Models | Seer | 2024-12 | RGB, ROB, TX | DP | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ |
| CoT-VLA | 2025-05 | RGB, ROB, TX | AR | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| WorldVLA | 2025-06 | RGB, ROB, TX | AR | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| DreamVLA | 2025-08 | RGB, ROB, TX | DM | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | |
| [ECoT VLA] (https://openreview.net/forum?id=S70MgnIA0v) | 2024-07 | RGB, TX | AR | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | |
| Policy Execution | PI-0 | 2024-10 | RGB, ROB, TX | FM | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ |
| PI-Fast | 2025-01 | RGB, ROB, TX | FM | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| PI-0.5 | 2025-04 | RGB, ROB, TX | FM | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| DisDiffVLA | 2025-08 | RGB, ROB, TX | DM | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | |
| Adaptation & Fine-Tuning | OpenVLA-OFT | 2025-02 | RGB, ROB, TX | DP | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ |
| ConRFT | 2025-04 | RGB, ROB, TX | DM | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | |
| RIPT-VLA | 2025-05 | RGB, ROB, TX | AR | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | |
| ControlVLA | 2025-06 | RGB, ROB, TX | DM | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
Notes
| Benchmark | Simulation Engine or Real-world Dataset | Embodiment | Data Size | RGB | Depth | Masks | Tool used | Multi-agents | Long-horizon |
|---|---|---|---|---|---|---|---|---|---|
| CALVIN | PyBullet | Franka Panda | 2400k | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ |
| Meta-world | MuJoCo | Sawyer | - | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| RLBench | CoppeliaSim | Franka Panda | - | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ |
| VIMAbench | PyBullet | UR5 | 650k | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ |
| LoHoRavens | PyBullet | UR5 | 15k | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ |
| ARNOLD | NVIDIA Omniverse | Framka Panda | 10k | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ |
| RoboGen | PyBullet | Multiple | - | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ |
| LIBERO | MuJoCo | Franka Panda | 6.5k | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ |
| Open X-Embodiment | Real-world Dataset | Multiple | 2419k | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ |
| DROID | Real-world Dataset | Franka Panda | 76k | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ |
| Galaxea Open-world | Real-world Dataset | Galaxea R1 Lite | 100k | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ |
| Models | Years | Benchmark | Simulation Engine | Language Module | Perception Module | Real world experiments | FMs | RL | IL | MP |
|---|---|---|---|---|---|---|---|---|---|---|
| IPRO | 2018 | # | - | LSTM | CNN | ✅ | ❌ | ❌ | ❌ | ✅ |
| MaestROB | 2018 | # | - | IBM Watson | Artoolkit | ✅ | ❌ | ❌ | ❌ | ✅ |
| exePlan | 2018 | # | - | coreNLP | * | ✅ | ❌ | ❌ | ❌ | ✅ |
| TLC | 2018 | # | - | CCG | * | ✅ | ❌ | ❌ | ✅ | ❌ |
| Cut&recombine | 2019 | # | - | Parser | * | ✅ | ❌ | ❌ | ❌ | ✅ |
| DREAMCELL | 2019 | # | - | LSTM | * | ❌ | ❌ | ❌ | ✅ | ❌ |
| ICR | 2019 | # | - | Parser, DCG | YOLO9000 | ✅ | ❌ | ❌ | ❌ | ✅ |
| GroundedDA | 2019 | # | - | CCG | RANSAC | ✅ | ❌ | ❌ | ❌ | ✅ |
| MEC | 2020 | # | - | Parser, ADCG | Mask RCNN | ✅ | ❌ | ❌ | ❌ | ✅ |
| LMCR | 2020 | # | - | RNN | Mask RCNN | ✅ | ❌ | ❌ | ❌ | ✅ |
| PixL2R | 2020 | Meta-World | MuJoCo | LSTM | CNN | ❌ | ❌ | ✅ | ❌ | ❌ |
| Concept2Robot | 2020 | # | PyBullet | BERT | ResNet-18 | ❌ | ❌ | ❌ | ✅ | ❌ |
| LanguagePolicy | 2020 | # | CoppeliaSim | GLoVe | Faster RCNN | ❌ | ❌ | ❌ | ✅ | ❌ |
| LOReL | 2021 | Meta-World | MuJoCo | distillBERT | CNN | ✅ | ❌ | ✅ | ❌ | ❌ |
| CARE | 2021 | Meta-World | MuJoCo | RoBERTa | * | ❌ | ✅ | ✅ | ❌ | ❌ |
| MCIL | 2021 | # | MuJoCo | MUSE | CNN | ❌ | ❌ | ❌ | ✅ | ❌ |
| BC-Z | 2021 | # | - | MUSE | ResNet18 | ✅ | ❌ | ❌ | ✅ | ❌ |
| CLIPort | 2021 | # | PyBullet | CLIP | CLIP | ✅ | ❌ | ❌ | ✅ | ❌ |
| LanCon-Learn | 2022 | Meta-World | MuJoCo | GLoVe | * | ❌ | ❌ | ✅ | ✅ | ❌ |
| MILLION | 2022 | Meta-World | MuJoCo | GLoVe | * | ✅ | ❌ | ✅ | ❌ | ❌ |
| PaLM-SayCan | 2022 | # | - | PaLM | ViLD | ✅ | ✅ | ✅ | ✅ | ❌ |
| ATLA | 2022 | # | PyBullet | BERT-Tiny | CNN | ❌ | ✅ | ✅ | ❌ | ❌ |
| HULC | 2022 | CALVIN | PyBullet | MiniLM-L3-v2 | CNN | ❌ | ❌ | ❌ | ✅ | ❌ |
| PerAct | 2022 | RLbench | CoppelaSim | CLIP | ViT | ✅ | ❌ | ❌ | ✅ | ❌ |
| RT-1 | 2022 | # | - | USE | EfficientNet-B3 | ✅ | ✅ | ❌ | ✅ | ❌ |
| LATTE | 2023 | # | CoppeliaSim | distillBERT, CLIP | CLIP | ✅ | ❌ | ❌ | ❌ | ✅ |
| DIAL | 2022 | # | - | CLIP | CLIP | ✅ | ✅ | ❌ | ✅ | ❌ |
| R3M | 2022 | # | - | distillBERT | ResNet18,34,50 | ✅ | ❌ | ❌ | ✅ | ❌ |
| Inner Monologue | 2022 | # | - | CLIP | CLIP | ✅ | ✅ | ❌ | ❌ | ✅ |
| NLMap | 2023 | # | - | CLIP | ViLD | ✅ | ✅ | ❌ | ✅ | ❌ |
| Code as Policies | 2023 | # | - | GPT3, Codex | ViLD | ✅ | ✅ | ❌ | ❌ | ✅ |
| Progprompt | 2023 | Virtualhome | Unity3D | GPT-3 | * | ✅ | ✅ | ❌ | ❌ | ✅ |
| Language2Reward | 2023 | # | MuJoCo MPC | GPT-4 | * | ✅ | ✅ | ✅ | ❌ | ❌ |
| LfS | 2023 | Meta-World | MuJoCo | Cons. Parser | * | ✅ | ❌ | ✅ | ❌ | ❌ |
| HULC++ | 2023 | CALVIN | PyBullet | MiniLM-L3-v2 | CNN | ✅ | ❌ | ❌ | ✅ | ❌ |
| ALOHA | 2023 | # | - | Transformer | CNN | ✅ | ❌ | ❌ | ✅ | ❌ |
| LEMMA | 2023 | LEMMA | NVIDIA Omniverse | CLIP | CLIP | ❌ | ❌ | ❌ | ✅ | ❌ |
| SPIL | 2023 | CALVIN | PyBullet | MiniLM-L3-v2 | CNN | ✅ | ❌ | ❌ | ✅ | ❌ |
| PaLM-E | 2023 | # | PyBullet | PaLM | ViT | ✅ | ✅ | ❌ | ✅ | ❌ |
| LAMP | 2023 | RLbench | CoppelaSim | ChatGPT | R3M | ❌ | ✅ | ✅ | ❌ | ❌ |
| MOO | 2023 | # | - | OWL-ViT | OWL-ViT | ✅ | ❌ | ❌ | ✅ | ❌ |
| Instruction2Act | 2023 | VIMAbench | PyBullet | ChatGPT | CLIP | ❌ | ✅ | ❌ | ❌ | ✅ |
| VoxPoser | 2023 | # | SAPIEN | GPT-4 | OWL-ViT | ✅ | ✅ | ❌ | ❌ | ✅ |
| SuccessVQA | 2023 | # | IA Playroom | Flamingo | Flamingo | ✅ | ✅ | ❌ | ✅ | ❌ |
| VIMA | 2023 | VIMAbench | PyBullet | T5 | ViT | ✅ | ✅ | ❌ | ✅ | ❌ |
| TidyBot | 2023 | # | - | GPT-3 | CLIP | ✅ | ✅ | ❌ | ❌ | ✅ |
| Text2Motion | 2023 | # | - | GPT-3, Codex | * | ✅ | ✅ | ✅ | ❌ | ❌ |
| LLM-GROP | 2023 | # | Gazebo | GPT-3 | * | ✅ | ✅ | ❌ | ❌ | ✅ |
| Scaling Up | 2023 | # | MuJoCo | CLIP, GPT-3 | ResNet-18 | ✅ | ✅ | ❌ | ✅ | ❌ |
| Socratic Models | 2023 | # | - | RoBERTa, GPT-3 | CLIP | ✅ | ✅ | ❌ | ❌ | ✅ |
| SayPlan | 2023 | # | - | GPT-4 | * | ✅ | ✅ | ❌ | ❌ | ✅ |
| RT-2 | 2023 | # | - | PaLI-X, PaLM-E | PaLI-X, PaLM-E | ✅ | ✅ | ❌ | ✅ | ❌ |
| KNOWNO | 2023 | # | PyBullet | PaLM-2L | * | ✅ | ✅ | ❌ | ❌ | ✅ |
| MDT | 2023 | CALVIN | PyBullet | CLIP | CLIP | ❌ | ❌ | ❌ | ✅ | ❌ |
| RT-Trajectory | 2023 | # | - | PaLM-E | EfficientNet-B3 | ✅ | ✅ | ❌ | ✅ | ❌ |
| SuSIE | 2023 | CALVIN | PyBullet | InstructPix2Pix(GPT3) | InstructPix2Pix | ✅ | ✅ | ❌ | ✅ | ❌ |
| Playfusion | 2023 | CALVIN | PyBullet | Sentence-bert | ResNet-18 | ✅ | ❌ | ❌ | ✅ | ❌ |
| ChainedDiffuser | 2023 | RLbench | CoppelaSim | CLIP | CLIP | ✅ | ❌ | ❌ | ✅ | ❌ |
| GNFactor | 2023 | RLbench | CoppelaSim | CLIP | NeRF | ✅ | ❌ | ❌ | ✅ | ❌ |
| StructDiffusion | 2023 | # | PyBullet | Sentence-bert | PCT | ✅ | ❌ | ❌ | ❌ | ✅ |
| PoCo | 2024 | Fleet-Tools | Drake | T5 | ResNet-18 | ✅ | ❌ | ❌ | ✅ | ❌ |
| DNAct | 2024 | RLbench | CoppelaSim | CLIP | NeRF, PointNext | ✅ | ❌ | ❌ | ✅ | ❌ |
| 3D Diffuser Actor | 2024 | CALVIN | PyBullet | CLIP | CLIP | ✅ | ❌ | ❌ | ✅ | ❌ |
| RoboFlamingo | 2024 | CALVIN | Pybullet | OpenFlamingo | OpenFlamingo | ❌ | ✅ | ❌ | ✅ | ❌ |
| OpenVLA | 2024 | Open X-Embodiment | - | Llama 2 7B | DINOv2 & SigLIP | ✅ | ✅ | ❌ | ✅ | ❌ |
| RT-X | 2024 | Open X-Embodiment | - | PaLI-X,PaLM-E | PaLI-X,PaLM-E | ✅ | ✅ | ❌ | ✅ | ❌ |
| PIVOT | 2024 | Open X-Embodiment | - | GPT-4, Gemini | GPT-4, Gemini | ✅ | ✅ | ❌ | ❌ | ✅ |
| RT-Hierarchy | 2024 | # | - | PaLI-X | PaLI-X | ✅ | ✅ | ❌ | ✅ | ❌ |
| 3D-VLA | 2024 | RL-Bench & CALVIN | CoppeliaSim & PyBullet | 3D-LLM | 3D-LLM | ❌ | ✅ | ❌ | ✅ | ❌ |
| Octo | 2024 | Open X-Embodiment | - | T5 | CNN | ✅ | ✅ | ❌ | ✅ | ❌ |
| ECoT | 2024 | BridgeData V2 | - | Llama 2 7B | DinoV2 & SigLIP | ✅ | ✅ | ❌ | ✅ | ❌ |
| LEGION | 2024 | Meta-World | MuJoCo | RoBERTa | * | ✅ | ❌ | ✅ | ❌ | ❌ |
| RACER | 2024 | RLbench | CoppelaSim | Llama3-llava-next-8B | LLaVA | ✅ | ✅ | ❌ | ✅ | ❌ |
| Ground4Act | 2024 | # | Gazebo | Transformer | ResNet101, BERT | ✅ | ❌ | ✅ | ❌ | ❌ |
| LOVM | 2024 | # | - | BiGRU | LOVM | ❌ | ❌ | ✅ | ❌ | ✅ |
| ECLAIR | 2024 | # | - | GPT-3-turbo | * | ✅ | ✅ | ✅ | ❌ | ❌ |
| PR2L | 2024 | MineDojo | HM3D | InstructBLIP | InstructBLIP | ✅ | ✅ | ✅ | ❌ | ❌ |
| AHA | 2024 | RLBench, ManiSkill | CoppeliaSim, SAPIEN | LLaMA-2-13B | CLIP | ❌ | ✅ | ✅ | ❌ | ✅ |
| KOI | 2024 | Meta-World, LIBERO | MuJoCo | GPT-4v | KOI | ✅ | ✅ | ❌ | ✅ | ✅ |
| GPT-4V(ISION) | 2024 | # | - | GPT-4 | GPT-4 | ✅ | ✅ | ❌ | ✅ | ❌ |
| HiRT | 2024 | Meta-World, Franka-Kitchen | MuJoCo, | InstructBLIP | CNN | ✅ | ✅ | ❌ | ✅ | ❌ |
| Sentinel | 2024 | # | - | GPT-4o | PointNet++ | ✅ | ✅ | ❌ | ❌ | ✅ |
| RoLD | 2024 | Open X-E, Robomimic, Meta-World | -, MuJoCo | DistilBERT | DistilBERT | ❌ | ❌ | ❌ | ❌ | ✅ |
| ITS | 2025 | * | - | LLaMA | A2C | ❌ | ✅ | ✅ | ❌ | ✅ |
| SIAMS | 2025 | Miniworld | Pyglet | LTL | CNN | ❌ | ✅ | ✅ | ❌ | ❌ | |
| CRTO | 2025 | Continual World | MuJoCo | ChatGPT | * | ❌ | ✅ | ✅ | ❌ | ✅ |
| LAMARL | 2025 | # | - | OpenAI | MADDPG | ✅ | ✅ | ✅ | ❌ | ✅ |
| ARCHIE | 2025 | # | - | GPT-4 | * | ✅ | ✅ | ✅ | ❌ | ✅ |
| RealBEF | 2025 | Meta-World | MuJoCo | ALBEF | CNN | ❌ | ✅ | ✅ | ❌ | ❌ |
| LLMRewardShaping | 2025 | Meta-World | MuJoCo | GPT-4 | * | ✅ | ✅ | ✅ | ❌ | ❌ |
| BOSS | 2025 | LIBERO | MuJoCo | OpenVLA | ResNet | ❌ | ✅ | ❌ | ✅ | ✅ |
| LAV-ACT | 2025 | # | MuJuCo | Voltron | Voltron | ✅ | ❌ | ❌ | ✅ | ❌ |
| TPM | 2025 | # | MuJuCo | GPT-4 | ResNet | ✅ | ✅ | ❌ | ✅ | ✅ |
| Mamba | 2025 | # | - | Mamba | Mamba | ✅ | ✅ | ❌ | ✅ | ✅ |
| TransformerPolicy | 2025 | CALVIN | PyBullet | Transformer | Sentence-BERT | ✅ | ❌ | ❌ | ✅ | ✅ |
| HierarchicalLCL | 2025 | CALVIN | PyBullet | OpenFlamingoM-3B | ViT | ❌ | ✅ | ❌ | ✅ | ✅ |
| BLADE | 2025 | CALVIN | PyBullet | GPT-4 | PCT | ✅ | ✅ | ❌ | ✅ | ✅ |
| LES6DPose | 2025 | # | Isaac Gym | GPT-4 | PointNet++ | ✅ | ✅ | ❌ | ❌ | |
| SafetyFilter | 2025 | # | - | GPT-4o | CLIP | ✅ | ✅ | ❌ | ❌ | ✅ |
| TARAD | 2025 | RLBench | CoppeliaSim | GPT-4o | CLIP | ✅ | ✅ | ❌ | ✅ | ✅ |
| DISCO | 2025 | CALVIN | PyBullet | GPT-4o | * | ✅ | ✅ | ❌ | ❌ | ✅ |
| TinyVLA | 2025 | Meta-World | MuJoCo | Pythia | MLP | ✅ | ✅ | ❌ | ❌ | ✅ |
| ASD-QR | 2025 | ScalingUp | MuJoCo | GPT3 | CLIP | ❌ | ✅ | ✅ | ❌ | ✅ |
| RDT-1B | 2025 | # | - | GPT-4-Turbo | T5-XXL | ✅ | ✅ | ❌ | ✅ | ❌ |
| GRAVMAD | 2025 | RLBench | CoppeliaSim | GPT-4o | CLIP | ✅ | ✅ | ❌ | ✅ | ✅ |
| GR-MG | 2025 | CALVIN | PyBullet | Transformer | T5-Base | ✅ | ❌ | ❌ | ❌ | ✅ |
| LEMMo-Plan | 2025 | # | - | GPT-4o | * | ✅ | ✅ | ❌ | ❌ | ✅ |
If you find this survey or repository useful, please consider citing:
@article{zhou2023language,
author = {Hongkuan Zhou and
Xiangtong Yao and
Oier Mees and
Yuan Meng and
Ted Xiao and
Yonatan Bisk and
Jean Oh and
Edward Johns and
Mohit Shridhar and
Dhruv Shah and
Jesse Thomason and
Kai Huang and
Joyce Chai and
Zhenshan Bing and
Alois Knoll},
title = {Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation},
journal = {CoRR},
volume = {abs/2312.10807},
year = {2023},
url = {https://doi.org/10.48550/arXiv.2312.10807}
}
27 commits
A curated, taxonomy-driven collection of language-conditioned robot manipulation papers, code, simulators, and benchmarks – tracking the literature behind
“Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation” (arXiv:2312.10807).
If you find this repo useful, please:
so more people can discover and build on this survey!
This repository is built around the survey:
Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation
Hongkuan Zhou, Xiangtong Yao, Oier Mees, Yuan Meng, Ted Xiao, Yonatan Bisk, Jean Oh, Edward Johns, Mohit Shridhar, Dhruv Shah, Jesse Thomason, Kai Huang, Joyce Chai, Zhenshan Bing, Alois Knoll
The structure of the survey organizes methods by the role language plays in the system.
At a high level, language can:
Evaluate what the robot is doing
-> Language for state evaluation
Language becomes a reward, cost, or scoring function, used to measure task progress, preferences, or goal satisfaction.
Specify how the robot should act
-> Language as a policy condition (Sec. 5)
Language is fed directly into the policy, shaping the action distribution at each step (e.g., language-conditioned RL, BC, diffusion policies).
Help the robot think and plan
-> Language for cognitive planning and reasoning (Sec. 6)
Language is used as an internal reasoning medium: planning, decomposition, querying knowledge bases, or manipulating symbolic structures.
Below, we briefly summarize each role and show how it maps to the sections in this repo.
Hierarchical understanding in robotic manipulation: A knowledge-based framework [paper]
Semantic Grasping Via a Knowledge Graph of Robotic Manipulation: A Graph Representation Learning Approach [paper]
Knowledge Acquisition and Completion for Long-Term Human-Robot Interactions using Knowledge Graph Embedding [paper]
Tell me dave: Context-sensitive grounding of natural language to manipulation instructions [paper]
Neuro-symbolic procedural planning with commonsense prompting [paper]
Reinforcement Learning Based Navigation with Semantic Knowledge of Indoor Environments [paper]
Learning Neuro-Symbolic Skills for Bilevel Planning [[paper]](Learning Neuro-Symbolic Skills for Bilevel Planning)
Learning Neuro-symbolic Programs for Language Guided Robot Manipulation [paper] [code]
Long-term robot manipulation task planning with scene graph and semantic knowledge [paper]
Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition [paper] [code]
ProgPrompt: program generation for situated robot task planning using large language models [paper]
Data-Agnostic Robotic Long-Horizon Manipulation with Vision-Language-Guided Closed-Loop Feedback [paper] [code]
Growing with Your Embodied Agent: A Human-in-the-Loop Lifelong Code Generation Framework for Long-Horizon Manipulation Skills [paper] [code]
| Optimization (Direction) | Article | Time | Observation | Action Generation | CoT | FP | MEM | MD | Pretraining CE | Scenarios MS | Scenarios RW | Execution CE |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Data Source Augmentation | EgoVLA | 2025-07 | RGB, ROB, TX | DP | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
| H-RDT | 2025-08 | RGB, ROB, TX | FM | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| Shortcut | 2025-08 | RGB, TX | - | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | |
| Spatial Understanding | SpatialVLA | 2025-01 | RGB, ROB, TX | AR | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ |
| PointVLA | 2025-05 | RGBD, ROB, TX | DM | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| BridgeVLA | 2025-06 | RGB, ROB, TX | DP | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | |
| GeoVLA | 2025-08 | RGBD, ROB, TX | DM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | |
| Multimodal Sensing & Fusion | VTLA | 2025-05 | RGB, TX | AR | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ |
| Tactile-VLA | 2025-07 | RGB, ROB, TX | FM | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | |
| OmniVTLA | 2025-08 | RGB, ROB, TX | FM | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ | |
| ForceVLA | 2025-09 | RGBD, ROB, TX | FM | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | |
| FuSe VLA | 2025-01 | RGBD, ROB, TX | AR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | |
| Long-horizon task solving | LoHoVLA | 2025-05 | RGB, ROB, TX | AR | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ |
| Long-VLA | 2025-08 | RGB, TX | DM | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | |
| DexVLA | 2025-08 | RGB, ROB, TX | DM | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| MemoryVLA | 2025-08 | RGB, TX | DM | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| DiffusionVLA | 2024-12 | RGBD, TX | DM + AR | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | |
| Knowledge Preserving | ChatVLA | 2025-02 | RGB, TX | DM | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ |
| ChatVLA-2 | 2025-05 | RGB, TX | DM | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | |
| Insulating | 2025-05 | RGB, ROB, TX | FM + AR | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| InstructVLA | 2025-07 | RGB, ROB, TX | FM | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | |
| GR-3 | 2025-07 | RGB, ROB, TX | FM | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | |
| Reasoning & World Models | Seer | 2024-12 | RGB, ROB, TX | DP | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ |
| CoT-VLA | 2025-05 | RGB, ROB, TX | AR | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| WorldVLA | 2025-06 | RGB, ROB, TX | AR | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| DreamVLA | 2025-08 | RGB, ROB, TX | DM | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | |
| [ECoT VLA] (https://openreview.net/forum?id=S70MgnIA0v) | 2024-07 | RGB, TX | AR | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | |
| Policy Execution | PI-0 | 2024-10 | RGB, ROB, TX | FM | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ |
| PI-Fast | 2025-01 | RGB, ROB, TX | FM | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| PI-0.5 | 2025-04 | RGB, ROB, TX | FM | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| DisDiffVLA | 2025-08 | RGB, ROB, TX | DM | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | |
| Adaptation & Fine-Tuning | OpenVLA-OFT | 2025-02 | RGB, ROB, TX | DP | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ |
| ConRFT | 2025-04 | RGB, ROB, TX | DM | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | |
| RIPT-VLA | 2025-05 | RGB, ROB, TX | AR | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | |
| ControlVLA | 2025-06 | RGB, ROB, TX | DM | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
Notes
| Benchmark | Simulation Engine or Real-world Dataset | Embodiment | Data Size | RGB | Depth | Masks | Tool used | Multi-agents | Long-horizon |
|---|---|---|---|---|---|---|---|---|---|
| CALVIN | PyBullet | Franka Panda | 2400k | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ |
| Meta-world | MuJoCo | Sawyer | - | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| RLBench | CoppeliaSim | Franka Panda | - | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ |
| VIMAbench | PyBullet | UR5 | 650k | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ |
| LoHoRavens | PyBullet | UR5 | 15k | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ |
| ARNOLD | NVIDIA Omniverse | Framka Panda | 10k | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ |
| RoboGen | PyBullet | Multiple | - | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ |
| LIBERO | MuJoCo | Franka Panda | 6.5k | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ |
| Open X-Embodiment | Real-world Dataset | Multiple | 2419k | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ |
| DROID | Real-world Dataset | Franka Panda | 76k | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ |
| Galaxea Open-world | Real-world Dataset | Galaxea R1 Lite | 100k | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ |
| Models | Years | Benchmark | Simulation Engine | Language Module | Perception Module | Real world experiments | FMs | RL | IL | MP |
|---|---|---|---|---|---|---|---|---|---|---|
| IPRO | 2018 | # | - | LSTM | CNN | ✅ | ❌ | ❌ | ❌ | ✅ |
| MaestROB | 2018 | # | - | IBM Watson | Artoolkit | ✅ | ❌ | ❌ | ❌ | ✅ |
| exePlan | 2018 | # | - | coreNLP | * | ✅ | ❌ | ❌ | ❌ | ✅ |
| TLC | 2018 | # | - | CCG | * | ✅ | ❌ | ❌ | ✅ | ❌ |
| Cut&recombine | 2019 | # | - | Parser | * | ✅ | ❌ | ❌ | ❌ | ✅ |
| DREAMCELL | 2019 | # | - | LSTM | * | ❌ | ❌ | ❌ | ✅ | ❌ |
| ICR | 2019 | # | - | Parser, DCG | YOLO9000 | ✅ | ❌ | ❌ | ❌ | ✅ |
| GroundedDA | 2019 | # | - | CCG | RANSAC | ✅ | ❌ | ❌ | ❌ | ✅ |
| MEC | 2020 | # | - | Parser, ADCG | Mask RCNN | ✅ | ❌ | ❌ | ❌ | ✅ |
| LMCR | 2020 | # | - | RNN | Mask RCNN | ✅ | ❌ | ❌ | ❌ | ✅ |
| PixL2R | 2020 | Meta-World | MuJoCo | LSTM | CNN | ❌ | ❌ | ✅ | ❌ | ❌ |
| Concept2Robot | 2020 | # | PyBullet | BERT | ResNet-18 | ❌ | ❌ | ❌ | ✅ | ❌ |
| LanguagePolicy | 2020 | # | CoppeliaSim | GLoVe | Faster RCNN | ❌ | ❌ | ❌ | ✅ | ❌ |
| LOReL | 2021 | Meta-World | MuJoCo | distillBERT | CNN | ✅ | ❌ | ✅ | ❌ | ❌ |
| CARE | 2021 | Meta-World | MuJoCo | RoBERTa | * | ❌ | ✅ | ✅ | ❌ | ❌ |
| MCIL | 2021 | # | MuJoCo | MUSE | CNN | ❌ | ❌ | ❌ | ✅ | ❌ |
| BC-Z | 2021 | # | - | MUSE | ResNet18 | ✅ | ❌ | ❌ | ✅ | ❌ |
| CLIPort | 2021 | # | PyBullet | CLIP | CLIP | ✅ | ❌ | ❌ | ✅ | ❌ |
| LanCon-Learn | 2022 | Meta-World | MuJoCo | GLoVe | * | ❌ | ❌ | ✅ | ✅ | ❌ |
| MILLION | 2022 | Meta-World | MuJoCo | GLoVe | * | ✅ | ❌ | ✅ | ❌ | ❌ |
| PaLM-SayCan | 2022 | # | - | PaLM | ViLD | ✅ | ✅ | ✅ | ✅ | ❌ |
| ATLA | 2022 | # | PyBullet | BERT-Tiny | CNN | ❌ | ✅ | ✅ | ❌ | ❌ |
| HULC | 2022 | CALVIN | PyBullet | MiniLM-L3-v2 | CNN | ❌ | ❌ | ❌ | ✅ | ❌ |
| PerAct | 2022 | RLbench | CoppelaSim | CLIP | ViT | ✅ | ❌ | ❌ | ✅ | ❌ |
| RT-1 | 2022 | # | - | USE | EfficientNet-B3 | ✅ | ✅ | ❌ | ✅ | ❌ |
| LATTE | 2023 | # | CoppeliaSim | distillBERT, CLIP | CLIP | ✅ | ❌ | ❌ | ❌ | ✅ |
| DIAL | 2022 | # | - | CLIP | CLIP | ✅ | ✅ | ❌ | ✅ | ❌ |
| R3M | 2022 | # | - | distillBERT | ResNet18,34,50 | ✅ | ❌ | ❌ | ✅ | ❌ |
| Inner Monologue | 2022 | # | - | CLIP | CLIP | ✅ | ✅ | ❌ | ❌ | ✅ |
| NLMap | 2023 | # | - | CLIP | ViLD | ✅ | ✅ | ❌ | ✅ | ❌ |
| Code as Policies | 2023 | # | - | GPT3, Codex | ViLD | ✅ | ✅ | ❌ | ❌ | ✅ |
| Progprompt | 2023 | Virtualhome | Unity3D | GPT-3 | * | ✅ | ✅ | ❌ | ❌ | ✅ |
| Language2Reward | 2023 | # | MuJoCo MPC | GPT-4 | * | ✅ | ✅ | ✅ | ❌ | ❌ |
| LfS | 2023 | Meta-World | MuJoCo | Cons. Parser | * | ✅ | ❌ | ✅ | ❌ | ❌ |
| HULC++ | 2023 | CALVIN | PyBullet | MiniLM-L3-v2 | CNN | ✅ | ❌ | ❌ | ✅ | ❌ |
| ALOHA | 2023 | # | - | Transformer | CNN | ✅ | ❌ | ❌ | ✅ | ❌ |
| LEMMA | 2023 | LEMMA | NVIDIA Omniverse | CLIP | CLIP | ❌ | ❌ | ❌ | ✅ | ❌ |
| SPIL | 2023 | CALVIN | PyBullet | MiniLM-L3-v2 | CNN | ✅ | ❌ | ❌ | ✅ | ❌ |
| PaLM-E | 2023 | # | PyBullet | PaLM | ViT | ✅ | ✅ | ❌ | ✅ | ❌ |
| LAMP | 2023 | RLbench | CoppelaSim | ChatGPT | R3M | ❌ | ✅ | ✅ | ❌ | ❌ |
| MOO | 2023 | # | - | OWL-ViT | OWL-ViT | ✅ | ❌ | ❌ | ✅ | ❌ |
| Instruction2Act | 2023 | VIMAbench | PyBullet | ChatGPT | CLIP | ❌ | ✅ | ❌ | ❌ | ✅ |
| VoxPoser | 2023 | # | SAPIEN | GPT-4 | OWL-ViT | ✅ | ✅ | ❌ | ❌ | ✅ |
| SuccessVQA | 2023 | # | IA Playroom | Flamingo | Flamingo | ✅ | ✅ | ❌ | ✅ | ❌ |
| VIMA | 2023 | VIMAbench | PyBullet | T5 | ViT | ✅ | ✅ | ❌ | ✅ | ❌ |
| TidyBot | 2023 | # | - | GPT-3 | CLIP | ✅ | ✅ | ❌ | ❌ | ✅ |
| Text2Motion | 2023 | # | - | GPT-3, Codex | * | ✅ | ✅ | ✅ | ❌ | ❌ |
| LLM-GROP | 2023 | # | Gazebo | GPT-3 | * | ✅ | ✅ | ❌ | ❌ | ✅ |
| Scaling Up | 2023 | # | MuJoCo | CLIP, GPT-3 | ResNet-18 | ✅ | ✅ | ❌ | ✅ | ❌ |
| Socratic Models | 2023 | # | - | RoBERTa, GPT-3 | CLIP | ✅ | ✅ | ❌ | ❌ | ✅ |
| SayPlan | 2023 | # | - | GPT-4 | * | ✅ | ✅ | ❌ | ❌ | ✅ |
| RT-2 | 2023 | # | - | PaLI-X, PaLM-E | PaLI-X, PaLM-E | ✅ | ✅ | ❌ | ✅ | ❌ |
| KNOWNO | 2023 | # | PyBullet | PaLM-2L | * | ✅ | ✅ | ❌ | ❌ | ✅ |
| MDT | 2023 | CALVIN | PyBullet | CLIP | CLIP | ❌ | ❌ | ❌ | ✅ | ❌ |
| RT-Trajectory | 2023 | # | - | PaLM-E | EfficientNet-B3 | ✅ | ✅ | ❌ | ✅ | ❌ |
| SuSIE | 2023 | CALVIN | PyBullet | InstructPix2Pix(GPT3) | InstructPix2Pix | ✅ | ✅ | ❌ | ✅ | ❌ |
| Playfusion | 2023 | CALVIN | PyBullet | Sentence-bert | ResNet-18 | ✅ | ❌ | ❌ | ✅ | ❌ |
| ChainedDiffuser | 2023 | RLbench | CoppelaSim | CLIP | CLIP | ✅ | ❌ | ❌ | ✅ | ❌ |
| GNFactor | 2023 | RLbench | CoppelaSim | CLIP | NeRF | ✅ | ❌ | ❌ | ✅ | ❌ |
| StructDiffusion | 2023 | # | PyBullet | Sentence-bert | PCT | ✅ | ❌ | ❌ | ❌ | ✅ |
| PoCo | 2024 | Fleet-Tools | Drake | T5 | ResNet-18 | ✅ | ❌ | ❌ | ✅ | ❌ |
| DNAct | 2024 | RLbench | CoppelaSim | CLIP | NeRF, PointNext | ✅ | ❌ | ❌ | ✅ | ❌ |
| 3D Diffuser Actor | 2024 | CALVIN | PyBullet | CLIP | CLIP | ✅ | ❌ | ❌ | ✅ | ❌ |
| RoboFlamingo | 2024 | CALVIN | Pybullet | OpenFlamingo | OpenFlamingo | ❌ | ✅ | ❌ | ✅ | ❌ |
| OpenVLA | 2024 | Open X-Embodiment | - | Llama 2 7B | DINOv2 & SigLIP | ✅ | ✅ | ❌ | ✅ | ❌ |
| RT-X | 2024 | Open X-Embodiment | - | PaLI-X,PaLM-E | PaLI-X,PaLM-E | ✅ | ✅ | ❌ | ✅ | ❌ |
| PIVOT | 2024 | Open X-Embodiment | - | GPT-4, Gemini | GPT-4, Gemini | ✅ | ✅ | ❌ | ❌ | ✅ |
| RT-Hierarchy | 2024 | # | - | PaLI-X | PaLI-X | ✅ | ✅ | ❌ | ✅ | ❌ |
| 3D-VLA | 2024 | RL-Bench & CALVIN | CoppeliaSim & PyBullet | 3D-LLM | 3D-LLM | ❌ | ✅ | ❌ | ✅ | ❌ |
| Octo | 2024 | Open X-Embodiment | - | T5 | CNN | ✅ | ✅ | ❌ | ✅ | ❌ |
| ECoT | 2024 | BridgeData V2 | - | Llama 2 7B | DinoV2 & SigLIP | ✅ | ✅ | ❌ | ✅ | ❌ |
| LEGION | 2024 | Meta-World | MuJoCo | RoBERTa | * | ✅ | ❌ | ✅ | ❌ | ❌ |
| RACER | 2024 | RLbench | CoppelaSim | Llama3-llava-next-8B | LLaVA | ✅ | ✅ | ❌ | ✅ | ❌ |
| Ground4Act | 2024 | # | Gazebo | Transformer | ResNet101, BERT | ✅ | ❌ | ✅ | ❌ | ❌ |
| LOVM | 2024 | # | - | BiGRU | LOVM | ❌ | ❌ | ✅ | ❌ | ✅ |
| ECLAIR | 2024 | # | - | GPT-3-turbo | * | ✅ | ✅ | ✅ | ❌ | ❌ |
| PR2L | 2024 | MineDojo | HM3D | InstructBLIP | InstructBLIP | ✅ | ✅ | ✅ | ❌ | ❌ |
| AHA | 2024 | RLBench, ManiSkill | CoppeliaSim, SAPIEN | LLaMA-2-13B | CLIP | ❌ | ✅ | ✅ | ❌ | ✅ |
| KOI | 2024 | Meta-World, LIBERO | MuJoCo | GPT-4v | KOI | ✅ | ✅ | ❌ | ✅ | ✅ |
| GPT-4V(ISION) | 2024 | # | - | GPT-4 | GPT-4 | ✅ | ✅ | ❌ | ✅ | ❌ |
| HiRT | 2024 | Meta-World, Franka-Kitchen | MuJoCo, | InstructBLIP | CNN | ✅ | ✅ | ❌ | ✅ | ❌ |
| Sentinel | 2024 | # | - | GPT-4o | PointNet++ | ✅ | ✅ | ❌ | ❌ | ✅ |
| RoLD | 2024 | Open X-E, Robomimic, Meta-World | -, MuJoCo | DistilBERT | DistilBERT | ❌ | ❌ | ❌ | ❌ | ✅ |
| ITS | 2025 | * | - | LLaMA | A2C | ❌ | ✅ | ✅ | ❌ | ✅ |
| SIAMS | 2025 | Miniworld | Pyglet | LTL | CNN | ❌ | ✅ | ✅ | ❌ | ❌ | |
| CRTO | 2025 | Continual World | MuJoCo | ChatGPT | * | ❌ | ✅ | ✅ | ❌ | ✅ |
| LAMARL | 2025 | # | - | OpenAI | MADDPG | ✅ | ✅ | ✅ | ❌ | ✅ |
| ARCHIE | 2025 | # | - | GPT-4 | * | ✅ | ✅ | ✅ | ❌ | ✅ |
| RealBEF | 2025 | Meta-World | MuJoCo | ALBEF | CNN | ❌ | ✅ | ✅ | ❌ | ❌ |
| LLMRewardShaping | 2025 | Meta-World | MuJoCo | GPT-4 | * | ✅ | ✅ | ✅ | ❌ | ❌ |
| BOSS | 2025 | LIBERO | MuJoCo | OpenVLA | ResNet | ❌ | ✅ | ❌ | ✅ | ✅ |
| LAV-ACT | 2025 | # | MuJuCo | Voltron | Voltron | ✅ | ❌ | ❌ | ✅ | ❌ |
| TPM | 2025 | # | MuJuCo | GPT-4 | ResNet | ✅ | ✅ | ❌ | ✅ | ✅ |
| Mamba | 2025 | # | - | Mamba | Mamba | ✅ | ✅ | ❌ | ✅ | ✅ |
| TransformerPolicy | 2025 | CALVIN | PyBullet | Transformer | Sentence-BERT | ✅ | ❌ | ❌ | ✅ | ✅ |
| HierarchicalLCL | 2025 | CALVIN | PyBullet | OpenFlamingoM-3B | ViT | ❌ | ✅ | ❌ | ✅ | ✅ |
| BLADE | 2025 | CALVIN | PyBullet | GPT-4 | PCT | ✅ | ✅ | ❌ | ✅ | ✅ |
| LES6DPose | 2025 | # | Isaac Gym | GPT-4 | PointNet++ | ✅ | ✅ | ❌ | ❌ | |
| SafetyFilter | 2025 | # | - | GPT-4o | CLIP | ✅ | ✅ | ❌ | ❌ | ✅ |
| TARAD | 2025 | RLBench | CoppeliaSim | GPT-4o | CLIP | ✅ | ✅ | ❌ | ✅ | ✅ |
| DISCO | 2025 | CALVIN | PyBullet | GPT-4o | * | ✅ | ✅ | ❌ | ❌ | ✅ |
| TinyVLA | 2025 | Meta-World | MuJoCo | Pythia | MLP | ✅ | ✅ | ❌ | ❌ | ✅ |
| ASD-QR | 2025 | ScalingUp | MuJoCo | GPT3 | CLIP | ❌ | ✅ | ✅ | ❌ | ✅ |
| RDT-1B | 2025 | # | - | GPT-4-Turbo | T5-XXL | ✅ | ✅ | ❌ | ✅ | ❌ |
| GRAVMAD | 2025 | RLBench | CoppeliaSim | GPT-4o | CLIP | ✅ | ✅ | ❌ | ✅ | ✅ |
| GR-MG | 2025 | CALVIN | PyBullet | Transformer | T5-Base | ✅ | ❌ | ❌ | ❌ | ✅ |
| LEMMo-Plan | 2025 | # | - | GPT-4o | * | ✅ | ✅ | ❌ | ❌ | ✅ |
If you find this survey or repository useful, please consider citing:
@article{zhou2023language,
author = {Hongkuan Zhou and
Xiangtong Yao and
Oier Mees and
Yuan Meng and
Ted Xiao and
Yonatan Bisk and
Jean Oh and
Edward Johns and
Mohit Shridhar and
Dhruv Shah and
Jesse Thomason and
Kai Huang and
Joyce Chai and
Zhenshan Bing and
Alois Knoll},
title = {Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation},
journal = {CoRR},
volume = {abs/2312.10807},
year = {2023},
url = {https://doi.org/10.48550/arXiv.2312.10807}
}
27 commits