Using VLM-based visual question answering to perceive scenes and control robots in MuJoCo simulation.
Python
6
27 commits
updated Oct 4, 2026
A research playground for language- and vision-driven robot control: robot manipulation where a vision-language model (VLM) either answers questions about the scene (VQA) or outputs actions directly (VLA).
The repo builds this up in layers, from low-level control through learning to large models: controllers (differential IK, null-space control) → planners (an LLM turns an instruction into subgoals) → learned policies (DAgger, PPO) → VLAs evaluated on standard benchmarks → simulated VQA/VLA data for training them.
The answer to a visual question can be a fact about the scene or the next action:
img = camera.capture()
answer = vlm.ask(img, "Where is the red cube?") # VQA: scene understanding -> structured state
action = vla.get_action(img, "pick up the red cube") # VLA: answer = action (e.g. 7-D EE delta + gripper)
robot.execute(action) # repeat until the task is done
| Area | Where | What |
|---|---|---|
| Classical control (MuJoCo) | src/, src/llm_vlm_robot_control/ | Franka pick-and-place with waypoints, UR5 differential IK with null-space obstacle avoidance (based on mjctrl) |
| LLM planning | src/llm_vlm_robot_control/llm_control.py | Qwen2.5-3B reads the instruction and scene state, returns JSON subgoals, and diff-IK executes them in MuJoCo |
| Imitation learning | src/BC/ | DAgger on 2D/3D reaching arms with state, vision and fused policies |
| Benchmarks | baseline/LIBERO/, baseline/CALVIN/ | Working env setups plus a pluggable get_action(image, instruction) rollout loop |
| LLM agents | baseline/ALFWorld/alfworld_eval/ | Zero-shot Qwen3 evaluation on ALFWorld (text env + AI2-THOR embodied demo) |
| VLA evaluation | baseline/Models/ | One Policy interface for OpenVLA, SmolVLA, pi0/pi0.5, RT-1, Octo, RT-2, CoT-VLA |
| Isaac Lab | rob_envs/isaac/ | Franka and H1 humanoid sims, PPO walking training, and scripted VQA/VLA dataset collection |
| VLM + decision model (MuJoCo) | rob_envs/MuJoCo/ | Franka pick-and-place where Qwen3-VL perceives and Jev (TypeSafe AI) makes typed decisions; qwen / jev / combined modes benchmarked |
| VLM closed-loop planning (MuJoCo) | rob_envs/MuJoCo/ | Multi-step 5-cube sorting planned by GPT-5.x or Qwen3-VL from side + wrist camera views, with memory of recent states |
Implements differential inverse kinematics with null space control on a UR5e arm in MuJoCo. The end-effector tracks a waypoint trajectory while avoiding a spherical obstacle, with joint velocities solved via a damped pseudoinverse Jacobian.
Minimal Isaac Lab 3.0 / Isaac Sim 6.1 examples in a separate isaac_venv (Python 3.12). See rob_envs/isaac/ for install and commands.
franka_sim.py --record).h1_walk.sh), or train PPO from scratch with live learning-curve plots (h1_train.sh).(image, instruction, action) per step, with actions also binned to 256 text tokens. It also saves ground-truth (image, question, answer) pairs about the scene. See rob_envs/isaac/vqa_data/.
VQA sample: start → grasp → lift, instruction "grasp the blue cube and lift it"
| Question / instruction | Answer |
|---|---|
| Where is the blue cube in the image? (pixel x, y) | (88, 110) |
| What is the 3D position of the red cube in meters? | (0.36, 0.18, 0.02) |
| Which cube is closest to the gripper? | blue |
| Which cube is on the left side of the image? | blue |
| grasp the blue cube and lift it (step 0 action) | [0.0006, -0.0042, -0.0090, 0, 0, 0, +1] → tokens 131 101 70 128 128 128 255 |
| grasp the blue cube and lift it (step 47 action) | [0, 0, 0, 0, 0, 0, -1] (close gripper) → tokens 128 128 128 128 128 128 0 |
A Franka Panda follows instructions such as "the grass-colored block belongs on the white mat". Jev is TypeSafe AI's text-only System 1 decision model, and here it makes the decisions:
Qwen3-VL-8B handles perception (2D points on the camera image, lifted to 3D with depth). In combined mode Qwen also acts as System 2, reasoning step by step only when Jev's confidence is below 0.7. A differential-IK controller executes the grasp.
Over 50 episodes per mode, Jev chose the right cube and mat every time, including paraphrases, negations and Qwen's habit of calling the purple mat "pink". It decided in a median of 0.17–0.25 s and never had to escalate to Qwen. The remaining task failures come from Qwen localizing a cube several cm off. See rob_envs/MuJoCo/ for the three modes, commands and the full comparison.
A Franka Panda sorts five cubes onto two mats from one instruction, for example "put every warm-colored cube on the purple mat and every cool-colored cube on the white mat". At every step the VLM (GPT-5.x through the OpenAI API, or Qwen3-VL-8B locally) gets one image with two views side by side: an eye-to-hand side view and an eye-in-hand wrist view. It also gets its last 3 states with the actions attempted from them. It points at the objects and re-plans, and the robot runs only the next action before observing again.
GPT-5.5 sorted all five cubes in 5 actions with no missed grasps ($0.32 per episode). Smaller models plan correctly but point too imprecisely to grasp a 4 cm cube. See rob_envs/MuJoCo/ for how it works, commands and results.
Baseline setup for the LIBERO manipulation benchmark, with a pluggable VLM/VLA rollout loop. See baseline/LIBERO/ for setup, scripts, and details.
Baseline setup for the CALVIN long-horizon, language-conditioned manipulation benchmark, with the same pluggable VLM/VLA rollout loop. See baseline/CALVIN/ for setup, scripts, and details.
Zero-shot evaluation of open-source LLMs (Qwen3-0.6B/1.7B/4B) on ALFWorld household tasks, with no training, no demonstrations and no expert plans. At each step the LLM picks one admissible command from the task, observation and recent history. See baseline/ALFWorld/alfworld_eval/ for setup and scripts, and RESULTS.md for the full per-task-type table.
Embodied (AI2-THOR) demo: the same zero-shot agent (Qwen3-4B) acting in AlfredThorEnv, ALFWorld's 3D AI2-THOR twin of the text games, shown from the robot's egocentric camera (left). Next to it is a text-env episode (Qwen3-1.7B) heating an egg and placing it (right).
Best result: Qwen3-1.7B with a richer zero-shot prompt reaches 14.8 avg. success on valid_unseen (strict prompt: 5.0; a reference paper reports 12.4 for the same base model).
A shared Policy interface plus wrappers for OpenVLA, SmolVLA, pi0/pi0.5, RT-1, Octo, RT-2, and CoT-VLA, evaluated on the LIBERO and CALVIN baselines above. OpenVLA (real 7B, LIBERO-finetuned checkpoint), SmolVLA, and RT-1 (a stateful tf_agents checkpoint with real Universal Sentence Encoder instruction embeddings) are actually validated with real weights, each in its own venv (vla_venv/vla_venv_lerobot/vla_venv_tf - different, conflicting framework/torch-version requirements); the rest are wired up with correct loading code but not executed here (a fourth framework, multi-GB checkpoints, or no public weights at all). See baseline/Models/ for the eval harness, per-model status, and sample results.
Using VLM-based visual question answering to perceive scenes and control robots in MuJoCo simulation.
Python
6
27 commits
updated Oct 4, 2026
A research playground for language- and vision-driven robot control: robot manipulation where a vision-language model (VLM) either answers questions about the scene (VQA) or outputs actions directly (VLA).
The repo builds this up in layers, from low-level control through learning to large models: controllers (differential IK, null-space control) → planners (an LLM turns an instruction into subgoals) → learned policies (DAgger, PPO) → VLAs evaluated on standard benchmarks → simulated VQA/VLA data for training them.
The answer to a visual question can be a fact about the scene or the next action:
img = camera.capture()
answer = vlm.ask(img, "Where is the red cube?") # VQA: scene understanding -> structured state
action = vla.get_action(img, "pick up the red cube") # VLA: answer = action (e.g. 7-D EE delta + gripper)
robot.execute(action) # repeat until the task is done
| Area | Where | What |
|---|---|---|
| Classical control (MuJoCo) | src/, src/llm_vlm_robot_control/ | Franka pick-and-place with waypoints, UR5 differential IK with null-space obstacle avoidance (based on mjctrl) |
| LLM planning | src/llm_vlm_robot_control/llm_control.py | Qwen2.5-3B reads the instruction and scene state, returns JSON subgoals, and diff-IK executes them in MuJoCo |
| Imitation learning | src/BC/ | DAgger on 2D/3D reaching arms with state, vision and fused policies |
| Benchmarks | baseline/LIBERO/, baseline/CALVIN/ | Working env setups plus a pluggable get_action(image, instruction) rollout loop |
| LLM agents | baseline/ALFWorld/alfworld_eval/ | Zero-shot Qwen3 evaluation on ALFWorld (text env + AI2-THOR embodied demo) |
| VLA evaluation | baseline/Models/ | One Policy interface for OpenVLA, SmolVLA, pi0/pi0.5, RT-1, Octo, RT-2, CoT-VLA |
| Isaac Lab | rob_envs/isaac/ | Franka and H1 humanoid sims, PPO walking training, and scripted VQA/VLA dataset collection |
| VLM + decision model (MuJoCo) | rob_envs/MuJoCo/ | Franka pick-and-place where Qwen3-VL perceives and Jev (TypeSafe AI) makes typed decisions; qwen / jev / combined modes benchmarked |
| VLM closed-loop planning (MuJoCo) | rob_envs/MuJoCo/ | Multi-step 5-cube sorting planned by GPT-5.x or Qwen3-VL from side + wrist camera views, with memory of recent states |
Implements differential inverse kinematics with null space control on a UR5e arm in MuJoCo. The end-effector tracks a waypoint trajectory while avoiding a spherical obstacle, with joint velocities solved via a damped pseudoinverse Jacobian.
Minimal Isaac Lab 3.0 / Isaac Sim 6.1 examples in a separate isaac_venv (Python 3.12). See rob_envs/isaac/ for install and commands.
franka_sim.py --record).h1_walk.sh), or train PPO from scratch with live learning-curve plots (h1_train.sh).(image, instruction, action) per step, with actions also binned to 256 text tokens. It also saves ground-truth (image, question, answer) pairs about the scene. See rob_envs/isaac/vqa_data/.
VQA sample: start → grasp → lift, instruction "grasp the blue cube and lift it"
| Question / instruction | Answer |
|---|---|
| Where is the blue cube in the image? (pixel x, y) | (88, 110) |
| What is the 3D position of the red cube in meters? | (0.36, 0.18, 0.02) |
| Which cube is closest to the gripper? | blue |
| Which cube is on the left side of the image? | blue |
| grasp the blue cube and lift it (step 0 action) | [0.0006, -0.0042, -0.0090, 0, 0, 0, +1] → tokens 131 101 70 128 128 128 255 |
| grasp the blue cube and lift it (step 47 action) | [0, 0, 0, 0, 0, 0, -1] (close gripper) → tokens 128 128 128 128 128 128 0 |
A Franka Panda follows instructions such as "the grass-colored block belongs on the white mat". Jev is TypeSafe AI's text-only System 1 decision model, and here it makes the decisions:
Qwen3-VL-8B handles perception (2D points on the camera image, lifted to 3D with depth). In combined mode Qwen also acts as System 2, reasoning step by step only when Jev's confidence is below 0.7. A differential-IK controller executes the grasp.
Over 50 episodes per mode, Jev chose the right cube and mat every time, including paraphrases, negations and Qwen's habit of calling the purple mat "pink". It decided in a median of 0.17–0.25 s and never had to escalate to Qwen. The remaining task failures come from Qwen localizing a cube several cm off. See rob_envs/MuJoCo/ for the three modes, commands and the full comparison.
A Franka Panda sorts five cubes onto two mats from one instruction, for example "put every warm-colored cube on the purple mat and every cool-colored cube on the white mat". At every step the VLM (GPT-5.x through the OpenAI API, or Qwen3-VL-8B locally) gets one image with two views side by side: an eye-to-hand side view and an eye-in-hand wrist view. It also gets its last 3 states with the actions attempted from them. It points at the objects and re-plans, and the robot runs only the next action before observing again.
GPT-5.5 sorted all five cubes in 5 actions with no missed grasps ($0.32 per episode). Smaller models plan correctly but point too imprecisely to grasp a 4 cm cube. See rob_envs/MuJoCo/ for how it works, commands and results.
Baseline setup for the LIBERO manipulation benchmark, with a pluggable VLM/VLA rollout loop. See baseline/LIBERO/ for setup, scripts, and details.
Baseline setup for the CALVIN long-horizon, language-conditioned manipulation benchmark, with the same pluggable VLM/VLA rollout loop. See baseline/CALVIN/ for setup, scripts, and details.
Zero-shot evaluation of open-source LLMs (Qwen3-0.6B/1.7B/4B) on ALFWorld household tasks, with no training, no demonstrations and no expert plans. At each step the LLM picks one admissible command from the task, observation and recent history. See baseline/ALFWorld/alfworld_eval/ for setup and scripts, and RESULTS.md for the full per-task-type table.
Embodied (AI2-THOR) demo: the same zero-shot agent (Qwen3-4B) acting in AlfredThorEnv, ALFWorld's 3D AI2-THOR twin of the text games, shown from the robot's egocentric camera (left). Next to it is a text-env episode (Qwen3-1.7B) heating an egg and placing it (right).
Best result: Qwen3-1.7B with a richer zero-shot prompt reaches 14.8 avg. success on valid_unseen (strict prompt: 5.0; a reference paper reports 12.4 for the same base model).
A shared Policy interface plus wrappers for OpenVLA, SmolVLA, pi0/pi0.5, RT-1, Octo, RT-2, and CoT-VLA, evaluated on the LIBERO and CALVIN baselines above. OpenVLA (real 7B, LIBERO-finetuned checkpoint), SmolVLA, and RT-1 (a stateful tf_agents checkpoint with real Universal Sentence Encoder instruction embeddings) are actually validated with real weights, each in its own venv (vla_venv/vla_venv_lerobot/vla_venv_tf - different, conflicting framework/torch-version requirements); the rest are wired up with correct loading code but not executed here (a fourth framework, multi-GB checkpoints, or no public weights at all). See baseline/Models/ for the eval harness, per-model status, and sample results.