🚀 Kuavo Data Challenge

✨ New features!

This is a branch under continuous development! It now supports:

• Depth imaging under ACT / Diffusion policy, each with its own fusion method for RGB vs depth imaging. For more details, see ACT and Diffusion
• Multi-GPU acceleration provided by Accelerate! See Multi-GPU acceleration for details.
• Latest Lerobot version 0.4.2 support! lerobot
• Frame alignment!
• Complete restructuring of the directories.
• ···

More to come:

• End-effector delta control support
• Extra Imitation Learning based algorithms!

🌟 Overview

This repository is developed based on Lerobot, combined with Leju Kuavo robot, providing complete example code for data format conversion (rosbag → parquet), Imitation Learning (IL) training, simulator testing, and real robot deployment verification.

✨ Features

• Data format conversion module (rosbag → Lerobot parquet)
• IL model training framework (diffusion policy, ACT)
• Mujoco simulation support
• Real robot verification and deployment

⚠️ Note: This repository does not yet support end-effector control; currently only joint angle control is available!

♻️ Requirements

• System: Ubuntu 20.04 recommended (if you are running 22.04 / 24.04 it's suggested to use Docker containers)
• Python: Python 3.10 recommended
• ROS: ROS Noetic + Kuavo Robot ROS patches (it's OK if installed in Docker container)
• Dependencies: Docker, NVIDIA CUDA Toolkit (if GPU acceleration is needed)

📦 Installation

1. OS Configuration

Ubuntu 20.04 + NVIDIA CUDA Toolkit + Docker is recommended.

sudo apt update
sudo apt upgrade -y
ubuntu-drivers devices
# Tested verfied version is 535, you can try newer versions (do not use server branch)
sudo apt install nvidia-driver-535
# Reboot the computer
sudo reboot
# Verify driver installation
nvidia-smi

sudo apt install curl
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.17.8-1
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.17.8-1 && sudo apt-get install -y nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

sudo apt update
sudo apt install git
sudo apt install docker.io
# Configure NVIDIA Runtime in Docker
nvidia-ctk
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
sudo docker info | grep -i runtime
# The output should include "nvidia" Runtime

2. ROS Environment Configuration

Both Kuavo Mujoco simulation and real robot operation are based on the ROS Noetic environment. Since the real Kuavo robot uses Ubuntu 20.04 + ROS Noetic (non-docker), it is recommended to directly install ROS Noetic. If ROS Noetic cannot be installed due to a higher Ubuntu version, Docker can be used.

wget http://fishros.com/install -O fishros && . fishros
# Menu selection: 5 Configure system sources → 2 Change sources and clean third-party sources → 1 Add ROS sources
wget http://fishros.com/install -O fishros && . fishros
# Menu selection: 1 One-click installation → 2 Install without changing sources → Select ROS1 Noetic Desktop

roscore  # Open a new terminal
rosrun turtlesim turtlesim_node  # Open a new terminal
rosrun turtlesim turtle_teleop_key  # Open a new terminal

sudo vim /etc/docker/daemon.json

{
    "registry-mirrors": [
        "https://docker.m.daocloud.io",
        "https://docker.imgdb.de",
        "https://docker-0.unsee.tech",
        "https://docker.hlmirror.com",
        "https://docker.1ms.run",
        "https://func.ink",
        "https://lispy.org",
        "https://docker.xiaogenban1993.com"
    ]
}

sudo systemctl daemon-reload && sudo systemctl restart docker

mkdir /path/to/save/docker/ros/image
cd /path/to/save/docker/ros/image
vim Dockerfile

FROM ubuntu:20.04

ENV DEBIAN_FRONTEND=noninteractive

RUN sed -i 's|http://archive.ubuntu.com/ubuntu/|http://mirrors.tuna.tsinghua.edu.cn/ubuntu/|g' /etc/apt/sources.list && \
    sed -i 's|http://security.ubuntu.com/ubuntu/|http://mirrors.tuna.tsinghua.edu.cn/ubuntu/|g' /etc/apt/sources.list

RUN apt-get update && apt-get install -y locales tzdata gnupg lsb-release
RUN locale-gen en_US.UTF-8
ENV LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_ALL=en_US.UTF-8

# Set ROS debian sources
RUN sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'

# Add ROS keys
RUN apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654

# Install ROS Noetic
# Set keyboard layout to Chinese if necessary
RUN apt-get update && \
    apt-get install -y keyboard-configuration apt-utils && \
    echo 'keyboard-configuration keyboard-configuration/layoutcode string cn' | debconf-set-selections && \
    echo 'keyboard-configuration keyboard-configuration/modelcode string pc105' | debconf-set-selections && \
    echo 'keyboard-configuration keyboard-configuration/variant string ' | debconf-set-selections && \
    apt-get install -y ros-noetic-desktop-full && \
    apt-get install -y python3-rosdep python3-rosinstall python3-rosinstall-generator python3-wstool build-essential && \
    rm -rf /var/lib/apt/lists/*

# Initialize rosdep
RUN rosdep init

sudo docker build -t ubt2004_ros_noetic .

sudo docker run -it --name ubuntu_ros_container ubt2004_ros_noetic /bin/bash
# Or GPU launch (recommended)
sudo docker run -it --gpus all --runtime nvidia --name ubuntu_ros_container ubt2004_ros_noetic /bin/bash
# Optional, mount local directory paths, etc.
# sudo docker run -it --gpus all --runtime nvidia --name ubuntu_ros_container -v /path/to/your/code:/root/code ubt2004_ros_noetic /bin/bash

sudo docker start ubuntu_ros_container
sudo docker exec -it ubuntu_ros_container /bin/bash

source /opt/ros/noetic/setup.bash
roscore

3. Clone Code

# SSH
git clone --depth=1 https://github.com/LejuRobotics/kuavo_data_challenge.git
# Or
# HTTPS
git clone --depth=1 https://github.com/LejuRobotics/kuavo_data_challenge.git

Update the lerobot submodule under third_party:

cd kuavo_data_challenge
git submodule init
git submodule update --recursive

4. Python Environment Configuration

Choose one of the following:

• Use conda (recommended):

conda create -n kdc_icra python=3.10
conda activate kdc_icra

• Use venv:

Install python3.10 first, then use venv to create a virtual environment:

⚠️ Warning: ppa:deadsnakes no longer provide packages for ubuntu20.04 after June 2025, the following installation method may not work anymore:

sudo apt update
sudo apt install -y software-properties-common
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install -y python3.10 python3.10-venv python3.10-dev

You may need to build from source as follows:

sudo apt update
sudo apt install -y build-essential libssl-dev zlib1g-dev libncurses5-dev libncursesw5-devlibreadline-dev libsqlite3-dev libgdbm-dev libdb5.3-dev libbz2-dev libexpat1-dev liblzma-dev tk-dev libffi-dev uuid-dev wget

wget https://www.python.org/ftp/python/3.10.18/Python-3.10.18.tgz
tar -xzf Python-3.10.18.tgz
cd Python-3.10.18
./configure --prefix=$HOME/python3.10 --enable-optimizations
make -j$(nproc)
sudo make install

Now create the venv environment:

python3.10 -m venv kdc_dev
source kdc_dev/bin/activate

Check and ensure correct installation:

python  # Check Python version, confirm output is 3.10.xxx (usually 3.10.18)
# Example output:
# Python 3.10.18 (main, Jun  5 2025, 13:14:17) [GCC 11.2.0] on linux
# Type "help", "copyright", "credits" or "license" for more information.
# >>> 

pip --version # Check pip version, confirm output shows pip for 3.10
# Example output: pip 25.1 from /path/to/your/env/python3.10/site-packages/pip (python 3.10)

Install dependencies:

pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple  # If youre located within mainland China, it is recommended to change the source first to speed up download and installation
# There is no need for you to run this if otherwise!

pip install -r requirements_ilcode.txt   # No ROS Noetic required, but only guarantees functionality of kuavo_train imitation learning training code. kuavo_data (data conversion) and kuavo_deploy (deployment code) both depend on ROS
# Or
pip install -r requirements_total.txt    # Ensure ROS Noetic is installed first (recommended)

After installation, double-check the lerobot version: Should be Version 0.4.2 as of November 2025.

pip show lerobot

If not, reset the lerobot repository:

cd third_party/lerobot
git fetch
git reset --hard origin/main
cd ../../

Retry pip install -r requirement_xx.txt to retry installation.

If you encounter ffmpeg or torchcodec errors when running:

conda install ffmpeg==6.1.1

# Or

# pip uninstall torchcodec

📨 Usage

1. Data Format Conversion

Convert Kuavo native rosbag data to parquet format usable by the Lerobot framework:

python kuavo_data/CvtRosbag2Lerobot.py \
  --config-path=../configs/data/ \
  --config-name=KuavoRosbag2Lerobot.yaml \
  rosbag.rosbag_dir=/path/to/rosbag \
  rosbag.lerobot_dir=/path/to/lerobot_data

Description:

• rosbag.rosbag_dir: Path to original rosbag data
• rosbag.lerobot_dir: Path to save converted lerobot-parquet data. A subfolder named lerobot is usually created in this directory
• configs/data/KuavoRosbag2Lerobot.yaml: Please review and select cameras to enable and whether to use depth images as needed

Or, you can set args in configs/data/KuavoRosbag2Lerobot.yaml

2. Imitation Learning Training

Use the converted data for imitation learning training:

python kuavo_train/train_policy.py \
  --config-path=../configs/policy/ \
  --config-name=diffusion_config.yaml \
  task=your_task_name \
  method=your_method_name \
  root=/path/to/lerobot_data/lerobot \
  training.batch_size=128 \
  policy_name=diffusion

Description:

• task: Custom task name (preferably corresponding to the task definition in data conversion), e.g., pick and place
• method: Custom method name, used to distinguish different training runs, e.g., diffusion_bs128_usedepth_nofuse, etc.
• root: Local path to training data. Note to include lerobot. Should correspond to the data conversion save path in step 1: /path/to/lerobot_data/lerobot
• training.batch_size: Batch size, can be adjusted according to GPU memory
• policy_name: Policy to use, used for policy instantiation. Currently supports diffusion and act
• For other parameters, please refer to the yaml file documentation. It is recommended to directly modify the yaml file to avoid command-line input errors

Or, you can set args in configs/policy/act_config/diffusion_config.yaml

2.1 IL Training with Multi-GPU Support

Double-check installation of Accelerate: pip install accelerate (Usually automatically installed with Lerobot)

# Configure the accelerate yaml according to your machines specs
vim configs/accelerate/accelerate_config.yaml
# After configuration, try this example:
accelerate launch --config_file configs/accelerate/accelerate_config.yaml kuavo_train/train_policy_with_accelerate.py  --config-path=../configs/policy --config-name=diffusion_config.yaml

3. Simulator Testing

After training is complete, you can start the Mujoco simulator and call the deployment code for evaluation:

a. Start Mujoco simulator: For details, see readme for simulator

b. Call deployment code

• Configuration files are located in ./configs/deploy/:

  • kuavo_env.yaml: Kuavo robot execution environment configuration, with env_name as Kuavo-Sim. Change other parameters such as obs_key_map as needed.

• Please review the yaml file and modify the # inference configs related parameters (model loading), etc.

• Start automated inference deployment:

  python kuavo_deploy/eval_kuavo.py
  

• Follow the instructions. Generally, Select 3 first, then provide the kuavo_env.yaml path (configs/deploy/kuavo_env.yaml). Finally, select "8. Auto-test model in simulation, execute eval_episodes times:". For details on this operation, see kuavo deploy

4. Real Robot Testing

Same steps as part a in step 3, change the configuration file kuavo_env.yaml's env_name as Kuavo-Real.

• PC deployment steps to be updated; For Orin deployment, please check: README_AGX_ORIN.md

• The log during testing is located at log/kuavo_deploy/kuavo_deploy.log, please check thoroughly.

📡 ROS Topic Description

Simulation Environment:

Real Robot Environment:

📁 Output Structure

outputs/
├── train/<task>/<method>/run_<timestamp>/   # Training models and parameters
├── eval/<task>/<method>/run_<timestamp>/    # Test logs and videos

📂 Code Structure

KUAVO_DATA_CHALLENGE/
├── configs/                # Configuration files
├── kuavo_data/             # Data processing and conversion module
├── kuavo_deploy/           # Deployment scripts (simulator/real robot)
├── kuavo_train/            # Imitation learning training code
├── lerobot_patches/        # Lerobot runtime patches
├── outputs/                # Models and results
├── third_party/            # Lerobot dependencies
└── requirements_xxx.txt    # Dependency lists
└── README.md               # Documentation

🐒 About lerobot_patches

This directory contains compatibility patches for Lerobot, with main features including:

• Extend FeatureType to support RGB and Depth images
• Customize compute_episode_stats and create_stats_buffers for statistical calculations of images and depth data, min, max, mean, std, etc.
• Modify dataset_to_policy_features to ensure correct mapping of Kuavo RGB + Depth FeatureType

If you need to use Lerobot-based custom designs such as depth data, new FeatureTypes, normalization methods, etc., you can add them yourself. When using, import at the very beginning of the entry script (such as kuavo_train/train_policy.py and other training file code):

import lerobot_patches.custom_patches  # Ensure custom patches are applied, DON'T REMOVE THIS LINE!

🙏 Acknowledgments

This project is extended based on Lerobot. Thanks to the HuggingFace team for developing the open-source robot learning framework, which provides an important foundation for this project.