1. 🌍 Genie Sim

Genie Sim is the simulation platform from AgiBot. It provides developers with a complete toolchain for environment reconstruction, scene generalization, data collection, and automated evaluation. Its core module, Genie Sim Benchmark is a standardized tool dedicated to establishing the most accurate and authoritative evaluation for embodied intelligence.

The platform integrates 3D reconstruction with visual generation to create a high-fidelity simulation environment. It pioneers LLM-driven technology to generate vast simulation scenes and evaluation configurations in minutes. The evaluation system covers 200+ tasks across 100,000+ scenarios to establish a comprehensive capability profile for models. Genie Sim also opens over 10,000 hours of synthetic data including real-world robot operation scenarios.

The platform will significantly accelerate model development, reduce reliance on physical hardware, and empower innovation in embodied intelligence. Simulation assets, dataset, and code are fully open source.

2. ✨ Features

• 🧞 geniesim CLI: one command-line entry point for the entire platform — docker lifecycle, ROS 2 workspace builds, health diagnosis (doctor), pure-Python wheel deploy, asset / scene bootstrap. Packaged as a standalone PEP 517 / PEP 621 wheel.
• 🤖 Agent-ready SKILLs: packaged SKILL.md recipes for benchmarking, inference checks, scene generation, asset search, and teleop — invokable directly by Claude Code / agentic coding agents, so the simulator can be driven without a human first reading the docs.
• ⚡ Genie Sim RT Engine (realtime, interactive): geniesim_ros brings physics, render, and the robot into ROS 2 natively — teleop, MoveIt, and ros2_control close the loop on one shared sim_time. Multiple physics backends (Isaac Sim PhysX, Newton-standalone) selectable per scene, with cloth + soft-body support on the Newton path.
• 🎨 High-Fidelity Sim-Ready Assets: 5,140 validated 3D assets covering five real-world operation fields: retail, industry, catering, home and office. ModelScope.
• 🛰️ 3DGS-based Reconstruction Pipeline: Integrate 3DGS-based reconstruction process with visual generative model to synthesize realistic simulation environment with high-precision meshes. ModelScope.
• 🌐 Genie Sim World: A multimodal spatial world model which generates photorealistic 3D world from diverse input types in minutes. GitHub.
• 💬 LLM-Driven Scene Generation: Natural language-driven generation and generalization which instantly generates diverse simulation scenes through conversational interaction.
• 📦 Large-Scale Synthetic Dataset: Over 10,000 hours open-source synthetic data across 200+ loco-manipulation tasks with multi-sensor streams, alongside multi-dimensional variations.
• 🛠️ Synthetic Data Generation: Efficient toolkit for data collection with error-recovery mechanism, supporting both low-latency teleoperation and automated data programming. ModelScope.
• 🧪 Robust and Diverse Benchmark: Provide 100,000+ simulation scenarios and use LLM to autonomously generate task instructions and evaluation configurations. Discrepancy between simulation and real-world test results is less than 10%.
• 🔍 VLM-based Auto-Evaluation System: Full-spectrum evaluation criteria to provide model's capability profile covering manipulation skills, cognitive comprehension and task complexity.
• 🚀 Zero-Shot Sim-to-Real Transfer: Model trained with our synthetic data exhibits zero-shot sim-to-real transfer capability with superior task success rate compared to model trained with real data.

3. 🚀 Quick start

3.1 🧞 CLI & 🤖 SKILLs — install, build the container, get a shell

flowchart LR
  pip(["📦 pip install<br/>geniesim_cli<br/><i>(host)</i>"]) ==> docker(["🐳 geniesim<br/>docker up<br/><i>(host)</i>"]) ==> auto(["⚙️ tier-1 peers<br/>auto-install<br/><i>(in-container)</i>"]) ==> shell(["💻 geniesim<br/>docker into<br/><i>(host)</i>"])
  classDef step fill:#dbeafe,stroke:#1e40af,color:#1e3a8a,font-weight:bold
  class pip,docker,auto,shell step
  linkStyle 0,1,2 stroke:#1e40af,stroke-width:2px

# === on the host ===
pip install -e source/geniesim_cli/      # standalone CLI wheel
geniesim docker build                    # build the Isaac Sim 5.1 + Jazzy image
geniesim docker up                       # boot the container; the entrypoint
                                         # editable-installs every tier-1 from
                                         # the bind-mounted workspace
geniesim docker into                     # interactive shell inside the container

# === inside the container — sanity-check the stack ===
geniesim status                          # per-distribution health probe
geniesim doctor                          # diagnose & repair (rosdep, env, …)
geniesim bootstrap                       # re-install tier-1 peers + opt-in tier-2

🤖 With an AI assistant — just ask

The repo ships AGENTS.md + per-package skills/<name>/SKILL.md as machine-readable signposts. Claude Code, Cursor, etc. follow them on their own — say "run the pnp benchmark against localhost:8999" or "launch the WBC scene with MoveIt" and the agent walks the trail.

📖 Without an AI assistant — follow the signposts

AGENTS.md (root, boot sequence) → source/AGENTS.md (module map) → source/<pkg>/AGENTS.md (architecture + commands) → source/<pkg>/skills/<name>/SKILL.md (copy-paste recipes). Full CLI verb table: .agent/geniesim_cli.md.

[!IMPORTANT] 🎨 Tier-1 peers come up automatically; tier-2 peers are opt-in. geniesim docker up's entrypoint editable-installs every tier-1 peer declared in source/geniesim/pyproject.toml. The asset library geniesim_assets is a tier-1 peer distributed out of band (sourced from HuggingFace / ModelScope), and editable-installed by the entrypoint so every other peer can find it.

Tier-2 peers (geniesim_teleop · geniesim_generator · geniesim_world) have heavy stacks (VR / LLM / CUDA-ML) and are not auto-installed. Install with pip install -e "source/geniesim/[teleop|generator|world|all]" — or run geniesim bootstrap for an interactive re-install of the whole tree.

3.2 🧪 Benchmark — debug locally, compete globally on RoboColiseum

flowchart LR
  list(["📋 benchmark<br/>list / categories"]) ==> probe(["🔌 check-<br/>inference"]) ==> run(["🧪 benchmark<br/>run · batch"]) ==> submit(["🏆 RoboColiseum<br/>leaderboard"])
  classDef step fill:#ede9fe,stroke:#5b21b6,color:#3b0764,font-weight:bold
  class list,probe,run,submit step
  linkStyle 0,1,2 stroke:#5b21b6,stroke-width:2px

The Genie Sim Benchmark is the engine behind RoboColiseum — and the same tool you use to get ready for it:

• 🏆 It's a competition. RoboColiseum ranks embodied-AI policies on four boards (instruction / robust / manip / spatial). The boards and their baseline scores are in the Genie Sim Benchmark Leaderboard below.
• 🧪 Debug locally → submit remotely. Run the benchmark tasks on your own machine to debug your policy against your own inference server, then launch a model evaluation on the official leaderboard with the same server — no blind submissions.
• 🤖 Agent-friendly one-click SKILLs. The whole competition pipeline — download datasets, fetch the π_0.5 baseline weights, run an example leaderboard service, submit & track results — ships under source/geniesim_benchmark/skills/robocoliseum/; just ask an AI agent (Claude Code, Cursor, …) to drive them.

Run it locally — cat source/geniesim_benchmark/USAGE.md for how to run a benchmark task on your own machine against your inference server (CLI verbs, common commands, and overrides).

3.3 ⚡ RT Engine — realtime, interactive ROS 2 simulation

flowchart LR
  build(["🔨 ros build<br/>dev"]) ==> scene(["🎬 launch<br/>scene × launcher"]) ==> sim(["⚡ RT Engine<br/>PhysX · Newton · mjwarp"]) ==> moveit(["🦾 MoveIt 2<br/>+ WBC RViz"])
  classDef step fill:#dcfce7,stroke:#15803d,color:#14532d,font-weight:bold
  class build,scene,sim,moveit step
  linkStyle 0,1,2 stroke:#15803d,stroke-width:2px

The RT Engine ships its own SKILLs that walk you through every step — workspace build → scene launch → MoveIt + WBC RViz → bringing your own robot → teleop → recording → debugging. Don't memorize ros2 launch flags — follow the skill:

# inside the container:
geniesim ros build dev
cat source/geniesim_ros/skills/build-workspace/SKILL.md    # `geniesim ros build dev`
cat source/geniesim_ros/skills/launch-scene/SKILL.md       # scene × launcher matrix
cat source/geniesim_ros/skills/moveit-wbc/SKILL.md         # MoveIt 2 + WBC RViz for Genie G2
cat source/geniesim_ros/skills/add-robot/SKILL.md          # bring your own URDF / xacro
cat source/geniesim_ros/skills/*/SKILL.md                  # and more

3.4 🌐 Generate a 3D world — geniesim_world

flowchart LR
  pano(["🖼️ Panorama<br/>(.png)"]) ==> da(["🔭 DA360<br/>depth"]) ==> sharp(["✨ SHARP<br/>Gaussians"]) ==> world(["🌍 3D world<br/>(PLY · Gaussians)"])
  classDef step fill:#ffedd5,stroke:#c2410c,color:#7c2d12,font-weight:bold
  class pano,da,sharp,world step
  linkStyle 0,1,2 stroke:#c2410c,stroke-width:2px

Turn an equirectangular panorama into an explorable photorealistic 3D world. (Loading the output into the RT Engine is a 🚧 W.I.P. — for now the generated PLY / Gaussians are consumed by geniesim_world's own tooling.)

cat source/geniesim_world/skills/generate-world/SKILL.md

3.5 🧱 Scene Reconstruction — scene_reconstruction

flowchart LR
  input(["📥 LiDAR and Images"]) ==> align(["📐 Pixel-level<br/> Alignment of Point Cloud and <br/>Camera Pose"]) ==> gs(["🌈 3DGS"]) ==> world(["🌍 3D world<br/>(PLY · Gaussians)"])
  classDef step fill:#e0f2fe,stroke:#0369a1,color:#0c4a6e,font-weight:bold
  class input,align,gs,world step
  linkStyle 0,1,2 stroke:#0369a1,stroke-width:2px

Reconstruct high-fidelity 3D scene assets from real-world LiDAR scans and camera images. The pipeline aligns point clouds with camera poses at pixel level, trains 3D Gaussian Splatting, and produces simulation-ready assets.

cat source/scene_reconstruction/AGENTS.md

See source/scene_reconstruction/README.md for the full pipeline details and AGENTS.md for the per-subsystem agent guides.

3.6 💬 Generate a scene from a prompt — geniesim_generator

flowchart LR
  nl(["💬 NL prompt<br/>'an apple on the table'"]) ==> assets(["🔍 search<br/>asset library"]) ==> dsl(["📝 Scene-Language<br/>LLM_RESULT.py"]) ==> usda(["🏗️ scene.usda<br/>+ layout graph"])
  classDef step fill:#ede9fe,stroke:#6d28d9,color:#3b0764,font-weight:bold
  class nl,assets,dsl,usda step
  linkStyle 0,1,2 stroke:#6d28d9,stroke-width:2px

Describe a scene in words; an LLM grounds it against the asset library, writes a Scene-Language DSL program, and the generator compiles it into scene.usda + a relationship graph. Drive it through the Open WebUI agent loop, or have Claude write the DSL directly (no WebUI / no MCP server). Embeddings run via a text-embedding API (no GPU) or a local VL model (GPU) — your pick at deploy time.

Generated scenes are written under source/geniesim_generator/src/benchmark/config/llm_task/<scene_id>/<run_id>/. If generated inside the Docker stack, the same bind-mounted files appear at /opt/geniesim_generator/src/benchmark/config/llm_task/<scene_id>/<run_id>/ in the container and at the source/... path on the host. Open the resulting scene.usda with Isaac Sim inside the Genie Sim container so its payload paths resolve as generated; if you open it from the host side, adjust the payload paths inside the USDA first.

cat source/geniesim_generator/skills/deploy-generator/SKILL.md    # install + start: MCP stack + Open WebUI, text vs VL embeddings
cat source/geniesim_generator/skills/search-assets/SKILL.md       # query the asset index by concept
cat source/geniesim_generator/skills/generate-scene/SKILL.md      # prompt → LLM_RESULT.py → scene.usda

See source/geniesim_ros/README.md for the engine overview, source/README.md for the full module map (every geniesim_* package with its docs + skills), and AGENTS.md for the per-subsystem agent guides.

3.7 🎮 Teleoperation — geniesim_teleop

flowchart LR
  up(["🐳 docker5.1 up"]) ==> collect(["🎮 autoteleop.sh<br/>VR collection"]) ==> keep(["💾 keep / discard<br/>(y / n)"]) ==> post(["📦 post-process<br/>→ HDF5 episodes"])
  classDef step fill:#fce7f3,stroke:#be185d,color:#831843,font-weight:bold
  class up,collect,keep,post step
  linkStyle 0,1,2 stroke:#be185d,stroke-width:2px

Collect demonstration data by teleoperating the robot with a Pico VR headset, then turn the recorded rosbags into HDF5 episodes. Once the image is built and the GUI container is up (geniesim docker5.1 up succeeds), the whole flow is two host-side scripts:

# 1. VR collection — press y (keep) / n (discard) when the collection is done:
./source/geniesim_teleop/scripts/autoteleop.sh

# 2. Post-process the recording into HDF5 episodes:
./source/geniesim_teleop/scripts/autoteleop_post_process.sh <task_name>

See source/geniesim_teleop/README.md for the controller button map, Pico setup, and task configuration.

3.8 📥 Data collection — data_collection

flowchart LR
  build(["🔨 docker build +<br/>autocollect build"]) ==> list(["📋 autocollect list<br/>pick a task"]) ==> run(["📥 autocollect run<br/>cuRobo auto-planning"]) ==> ep(["💾 episodes<br/>recording_data/"])
  classDef step fill:#e0f2fe,stroke:#0369a1,color:#0c4a6e,font-weight:bold
  class build,list,run,ep step
  linkStyle 0,1,2 stroke:#0369a1,stroke-width:2px

The automated counterpart to teleoperation: pre-orchestrated tasks are collected unattended using cuRobo trajectory planning — no human in the loop. Install the geniesim client and assets first (see § 3.1).

# 1. Build the GenieSim base image, then the data-collection image:
geniesim docker build         # → registry.agibot.com/genie-sim/geniesim3:latest (base)
geniesim autocollect build    # → registry.agibot.com/genie-sim/geniesim3-data-collection:latest

# 2. Discover tasks, then collect one (headless, unattended):
geniesim autocollect list --robot=g2 <substr>
geniesim autocollect run <TASK> --headless --standalone

Or let your AI assistant drive the whole run — point it at the skill and it follows the workflow (resolve task → check prereqs → launch → monitor & verify):

cat source/data_collection/skills/run-data-collection/SKILL.md

Episodes land in source/data_collection/recording_data/. See source/data_collection/README.md (and its AGENTS.md / skill) for flags, task configs, and the two-process server/client layout.

4. 📜 Roadmap & Updates

The arc of the platform — what shipped when, and what's coming next.

flowchart LR
  v2["<b>v2.x · 2025</b><br/>Challenge tasks<br/>2× sim speed"]
  v30["<b>v3.0 · Jan 2026</b><br/>Isaac Sim 5.1<br/>Genie G2 · 3DGS"]
  v31["<b>v3.1 · Apr 2026</b><br/>Genie Sim World<br/>RLinf"]
  v32["<b>v3.2 · Jun 2026</b><br/>🧞 CLI · 🤖 SKILLs<br/>⚡ RT Engine"]
  next["<b>Next</b><br/>Isaac Sim 6.0<br/>More Newton solvers<br/>Benchmark refactor"]
  v2 --> v30 --> v31 --> v32 --> next

  classDef done fill:#d1f0d8,stroke:#1f6f3d,color:#0b3b1f
  classDef current fill:#cfe2ff,stroke:#084298,color:#084298,stroke-width:2px
  classDef future fill:#fff3cd,stroke:#856404,color:#5c4400
  class v2,v30,v31 done
  class v32 current
  class next future

🎯 Next

• Upload all assets and dataset on Huggingface
• More benchmark task suites — deformable / soft-material manipulation, whole-body control (WBC), vision-language-navigation (VLN), and more
• Support more tasks and larger models for RLinf
• More Newton solvers — broaden the rigid + deformable solver in the Newton-standalone
• Refactor geniesim_benchmark into a benchmark layer on top of geniesim_ros (collapses the two parallel stacks — see .agent/geniesim_benchmark.md)
• Data recorder improvements (today: ros2 bag via the record-episode SKILL)
• Wire geniesim_world outputs into the RT Engine (load .ply / .gsp Gaussians from a scene_*.yaml)
• Isaac Sim 6.0

📜 Release log

[6/25/2026] v3.2.0 — Genie Sim RT Engine + Agentic CLI/SKILLs

• 🏆 RoboColiseum — opened RoboColiseum: submit a policy and get it scored on the official leaderboard. Added a new spatial board (alongside instruction / robust / manip), and the robust board now aggregates and scores tasks per perturbation type (instruction, robot pose, background, image quality, camera position). The full pipeline — download datasets, fetch baseline weights, run an example leaderboard service, submit & track — ships as agent-friendly one-click SKILLs under source/geniesim_benchmark/skills/robocoliseum/.
• 🧞 geniesim CLI — one command for docker, ROS 2 builds, bootstrap, status, doctor, deploy; bash/zsh completion; standalone PEP 517 / PEP 621 wheel.
• 🤖 Agent SKILLs — self-contained SKILL.md recipes (run-benchmark, check-inference, generate-scene, search-assets, deploy-generator, run-teleop) that Claude Code and other agents invoke to drive the simulator end-to-end.
• ⚡ Genie Sim RT Engine (realtime, interactive) — geniesim_ros ships physics + render + robot as a first-class ROS 2 node sharing one sim_time. Multiple physics backends (Isaac Sim PhysX, Isaac Sim Newton, Kit-free Newton-standalone) with cloth / soft-body on the Newton path.
• 🦾 Customizable robots — Genie G2 family (Tier 1, continuously maintained) with arm × gripper matrix, drop-in MoveIt 2 + WBC RViz, three IK plugins; reference URDF for Franka, UR5, Aloha, ARX, Agilex; bring-your-own via xacro + offline mesh-prep tools.
• 🎬 New demos — pick-and-place (scene_pnp_g2_op) and whole-body control (scene_wbc_g2_sp).
• 🛠️ Engine internals — AS3 asset layout (per-robot robot.usda + payloads/Physics/{physics,physx,mujoco}.usda); IsaacSim 5.1 + 6.0 share one CLI surface; fisheye camera + de-skewed rotary lidar.
• 🔗 Module dependency DAG — auto-generated from each peer's pyproject.toml, rendered in source/README.md; enforced by geniesim tool deps-dag in CI.

[4/8/2026] v3.1

• 🌐 Release Genie Sim World — a multimodal spatial world model for 3D world generation.
• 🧪 Update new benchmarks for instruction following, spatial understanding, manipulation skills, robustness, and sim2real.
• 🎓 Support human-in-the-loop and distributed reinforcement learning pipeline of RLinf.

[1/7/2026] v3.0

• 🆙 Update Isaac Sim to v5.1.0 and support RTX 50series graphic card.
• 🦾 Provide USD and URDF files of Agibot Genie G2 robot and support whole body control.
• 🛰️ Support 3DGS-based scene reconstruction and convert output to USD format for application in Isaac Sim.
• 📦 Release synthetic dataset and corresponding data collection pipeline.
• 💬 Add LLM-based features to generate scenarios, task instructions and evaluation configurations.

[7/14/2025] v2.2

• 📊 Provide detailed evaluation metrics for all Agibot World Challenge tasks.
• ⚙️ Add automatic evaluation script to run each task multiple times and record score of all steps.

[6/25/2025] v2.1

• ➕ Add 10 more manipulation tasks for Agibot World Challenge 2025 including all simulation assets.
• 📦 Open-source synthetic datasets for 10 manipulation tasks on Huggingface.
• 🔌 Integrate UniVLA policy and support model inference simulation evaluation.
• 🧮 Update IK solver sdk which supports cross-embodiment IK solving for other robots.
• ⚡ Optimize communication framework and improve simulation running speed by 2×.
• 🧪 Update automatic evaluation framework for more complicated long-range tasks.

5. 📚 Documentation

5.1 📖 Documentations

Please refer to these links to download assets and dataset:

• Assets
• Dataset

5.2 🏆 Genie Sim Benchmark Leaderboard

The four tables below are the RoboColiseum boards — instruction, robust, manip, spatial — reporting baseline scores for four reference models. The final GenieSim-Sim2Real table is not a competition board; it quantifies the simulation-to-reality fidelity of the benchmark. Each task is evaluated under a 2×2 design that crosses the training-data source (simulation vs. real) with the evaluation environment (simulation vs. real), yielding the sim-to-sim, real-to-sim, sim-to-real, and real-to-real conditions. The close agreement between corresponding simulated and physical success rates indicates that evaluation in Genie Sim is a faithful proxy for real-world evaluation.

Baseline model repositories:

GenieSim-Manipulation

GenieSim-Spatial

GenieSim-Sim2Real

^† Sim Data: 500~1500 episodes of simulation data. Real Data: 500 episodes of real-world data. All models are post-trained from the π_0.5 baseline.

5.3 💬 Support

5.4 📄 License and Citation

All the data and code within source/geniesim_* and source/data_collection are under Mozilla Public License 2.0 unless licensed specifically. The source/scene_reconstruction project contains code under multiple licenses, for complete and updated licensing details, please see the LICENSE files

Please consider citing our work either way below if it helps your research.

@misc{yin2026geniesim30,
  title={Genie Sim 3.0 : A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot},
  author={Chenghao Yin and Da Huang and Di Yang and Jichao Wang and Nanshu Zhao and Chen Xu and Wenjun Sun and Linjie Hou and Zhijun Li and Junhui Wu and Zhaobo Liu and Zhen Xiao and Sheng Zhang and Lei Bao and Rui Feng and Zhenquan Pang and Jiayu Li and Qian Wang and Maoqing Yao},
  year={2026},
  eprint={2601.02078},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2601.02078},
}

5.5 🔗 References

1. PDDL Parser (2020). Version 1.1. [Source code]. https://github.com/pucrs-automated-planning/pddl-parser.
2. BDDL. Version 1.x.x [Source code]. https://github.com/StanfordVL/bddl
3. CUROBO [Source code]. https://github.com/NVlabs/curobo
4. Isaac Lab [Source code]. https://github.com/isaac-sim/IsaacLab
5. Omni Gibson [Source code]. https://github.com/StanfordVL/OmniGibson
6. The Scene Language [Source code]. https://github.com/zzyunzhi/scene-language
7. COAL [Source code]. https://github.com/coal-library/coal
8. OCTOMAP [Source code]. https://github.com/OctoMap/octomap
9. PINOCCHIO [Source code]. https://github.com/stack-of-tasks/pinocchio
10. URDFDOM [Source code]. https://github.com/ros/urdfdom
11. LIBCCD [Source code]. https://github.com/danfis/libccd
12. LIBMINIZIP [Source code]. https://github.com/switch-st/libminizip
13. LIBODE [Source code]. https://github.com/markmbaum/libode
14. LIBURING [Source code]. https://github.com/axboe/liburing
15. MuJoCo [Source code]. https://github.com/google-deepmind/mujoco