UBRobot

UBRobot is an experimental robot navigation and manipulation stack for building embodied AI agents that can see, reason, plan, and act in the real world. The project combines LeRobot-style robot interfaces, RGB-D perception, vision-language reasoning, navigation policy services, teleoperation tools, and a Gradio chat UI for issuing high-level natural-language instructions.

The repository is a work in progress, but the current codebase is organized around two main capabilities:

• Mobile robot navigation from language instructions, using RealSense RGB-D observations, odometry, VLM grounding/reasoning, InternNav/InternVLA-style policy inference, LogoPlanner components, and MPC/PID control.
• Robot arm manipulation and teleoperation, especially for the AgileX Piper arm, with LeRobot-compatible observation/action APIs, CAN control through piper-sdk, camera integration, point-cloud perception, and recording/evaluation examples.

Features

• LeRobot-compatible robot abstractions for Piper, SO-101 follower, LeKiwi base, and Unitree Go2 experiments.
• RGB-D camera utilities for Intel RealSense and aligned color/depth observations.
• Cortex-orchestrated navigation on LeKiwi: plain-language chat input is planned by EMOS Cortex and executed through a guarded semantic NavigateToObject ROS 2 Action (see "Cortex-Orchestrated Navigation" below).
• Legacy natural-language navigation loop in Go2Manager (rollback/research path only), including instruction handling, policy-server calls, trajectory following, and annotated visual feedback.
• Flask services for navigation policy inference and vision-language reasoning.
• Gradio chat interface for text or microphone input, live robot observation display, and command routing.
• Teleoperation examples for Piper and SO-101 robots, including keyboard, gamepad, and networked workflows.
• ROS dependency workspace for Piper ROS, RTAB-Map/odometry, and related hardware integration.
• Assets for robot URDFs, meshes, camera/arm configuration, sample media, and local model checkpoints.

Repository Layout

.
|-- assets/                 # URDFs, meshes, icons, sample media, configs, model assets
|-- docs/                   # Setup notes for robots, datasets, and sensors
|-- examples/               # Teleoperation, recording, evaluation, and model-service examples
|-- ros_depends_ws/         # ROS/Catkin workspace for hardware and odometry dependencies
|-- src/
|   |-- chat_ui/            # Gradio/FastAPI user interface
|   |-- service/
|   |   |-- planning/       # InternVLA/InternNav navigation policy HTTP server
|   |   `-- reasoning/      # VLM reasoning, grasp planning, GraspNet/PointNet utilities
|   `-- ubrobot/
|       |-- cameras/        # RealSense, odometry, and camera utilities
|       |-- robots/         # Robot drivers/adapters and high-level robot manager
|       `-- teleoperators/  # Keyboard/gamepad teleoperation helpers
|-- third_party/            # Vendored external code, including GraspNet API
|-- install.sh              # Native dependency setup notes/script
|-- pyproject.toml          # Python package metadata and dependencies
`-- requirements.txt        # Pinned Python dependency set

Main Components

Robot Interfaces

The robot layer in src/ubrobot/robots provides hardware adapters and higher-level control code:

• piper/ implements a LeRobot-compatible Piper arm interface with joint observations, actions, optional gripper control, RealSense/OpenCV camera support, and Piper SDK integration.
• lekiwi/ contains the mobile-base interface used by the navigation manager.
• so101_follower/ contains SO-101 host/client/follower control code.
• unitree_go2_robot.py wraps basic Unitree Go2 sport-client actions.
• ubrobot.py contains Go2Manager, the current high-level agent loop for observations, language instructions, navigation planning, and base motion commands.

Planning And Reasoning Services

The project separates heavyweight model inference into HTTP services:

• src/service/planning/http_internvla_server.py starts a Flask server for InternVLA/InternNav navigation policy inference on port 5801.
• src/service/reasoning/http_reasoning_server.py starts a Flask server for VLM reasoning on port 5802.
• src/service/reasoning/ also includes grasp planning, GraspNet models, PointNet/KNN extensions, and dataset utilities.

Chat UI

src/chat_ui/app.py starts a Gradio interface served through FastAPI. It accepts text or microphone input, sends commands through ChatPipeline, and displays navigation/manipulation observations. By default it runs on port 7863 with the local TLS certificate files under assets/.

Cortex-Orchestrated Navigation (LeKiwi)

The primary navigation path no longer relies on keyword prefixes or a locally connected robot. Plain-language UI text is submitted unchanged to the EMOS Cortex Action, which plans and executes through one semantic capability:

Chat UI -> /cortex_input_command (Cortex Action)
        -> /ubrobot/navigation/navigate_to_object (semantic capability, owns command lease)
        -> Kompass /track_vision_target -> /navigation/raw_cmd_vel
        -> cmd_vel_guard (lease + freshness gate, 50 ms tick)
        -> /cmd_vel -> lekiwi-base-driver (final clamp, watchdog, torque lifecycle)

Safety properties enforced in software:

• Cortex can discover only the semantic NavigateToObject capability — never raw velocity, torque, or device controls.
• The capability server never publishes /cmd_vel; it owns a short-lived command lease (/navigation/command_lease, 250 ms freshness).
• cmd_vel_guard forwards clamped velocity only while both the lease and the raw command are fresh, and emits zeros otherwise; cancellation, timeout, client death, and capability loss all stop motion within 300 ms (validated in docs/validation/2026-07-30-cortex-navigation-mock.md).
• Only the LeKiwi driver container maps /dev/lekiwi-base and manages motor torque; planner credentials come from runtime environment variables only.

Rollback: the previous keyword-based path remains available behind an explicit switch — set UBROBOT_CHAT_BACKEND=legacy before starting the Chat UI to use Go2Manager.agent_response() (deprecated; research/rollback only). On the robot, the pre-Cortex vision_depth_follower recipe remains in the EMOS image: deploy without deploy/emos/compose.cortex-navigation.yaml to roll back, or pin a recorded image tag such as ubrobot/emos:jazzy-7a64982.

See emos.md for operations, docs/plans/2026-07-30-cortex-navigation.md and docs/plans/2026-07-30-cortex-ui-routing-and-hardware-gate.md for the implementation plans, and docs/plans/2026-07-30-cortex-navigation-hardware-validation.md for the (authorized-separately) real-hardware gate.

Installation

UBRobot is currently developed for Linux robot machines, especially Ubuntu 20.04 style environments with Python 3.10, CUDA-capable PyTorch, ROS, RealSense, and robot-specific SDKs.

Create and activate a Python environment:

conda create -y -n ubrobot python=3.10
conda activate ubrobot

Install FFmpeg in the environment:

conda install ffmpeg=7.1.1 -c conda-forge

Install PyTorch. For CUDA 12.8:

pip install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128

For CPU-only environments:

pip install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cpu

Install common Python dependencies and the project in editable mode:

pip install -r requirements.txt
pip install -e .

Some hardware paths also require native dependencies such as ROS, RealSense, Piper CAN setup, CycloneDDS, Unitree SDK2 Python, and TRAC-IK/KDL libraries. See install.sh, ros_depends_ws/README.md, docs/install_realsense.md, and the scripts under ros_depends_ws/ for the current setup notes.

Running

Start the navigation policy service:

bash start_policy_server.bash

Start the reasoning service:

bash start_reasoning_server.bash

Start the robot/chat UI stack on the robot machine:

bash ubrobot_startup.sh

The UI is served by src/chat_ui/app.py and defaults to:

https://0.0.0.0:7863

The startup script also sources ROS, activates Piper CAN support, sets CYCLONEDDS_HOME, and launches the Gradio/FastAPI application.

Examples

Useful entry points include:

• examples/piper/teleoperate.py for Piper arm teleoperation.
• examples/piper/record.py for recording Piper demonstrations.
• examples/piper/evaluate.py for evaluating a policy on Piper.
• examples/so101_to_so101/ for networked SO-101 teleoperation and recording.
• examples/lerobot_record.py and examples/lerobot_eval.py for LeRobot-style dataset/policy workflows.
• examples/internnav_demo.py and examples/http_internvla_client.py for navigation-model experiments.

Related Projects

• InternNav: Open platform for generalized navigation foundation models.
• Qwen3-VL: Vision-language foundation model family.
• LeRobot: Robot learning datasets, policies, and robot abstractions used as a design reference.
• Cosmos: Physical-reasoning vision-language models for embodied decision making.
• GraspNet API: Grasp representation, visualization, and evaluation utilities.

Status

This repository is actively evolving. Some scripts contain machine-specific paths, local model checkpoint paths, and hardware-specific assumptions. Treat the project as a research/development workspace rather than a packaged production release.

Code hygiene (2026-08-08):

• Production packages (chat_ui, robot_edge, ubrobot_contracts) are ruff- and mypy-clean; core pure-Python coverage is 86% with a CI gate.
• Legacy hardware-direct code (Go2Manager, unitree_go2_robot, arm_action) is isolated as rollback/deprecated paths; see docs/hardware/legacy-rollback.md and src/ubrobot/robots/README.md.
• CI enforces lint, tests (unittest), secret scanning, coverage, and a hardware safety contract on every PR; see .github/workflows/ci.yml.

TODO

• Add a system architecture diagram.
• Add Docker or reproducible environment setup.
• Normalize machine-specific paths and configuration.
• Expand hardware setup documentation.