RoboML is an aggregator package written for prototyping and deploying open source ML models for robotics
11
stars
137
commits
Python
primary language
Sep 3, 2026
updated
README
RoboML is an aggregator package for quickly deploying open-source ML models for robots. It supports three main use cases:
- Rapid deployment of general-purpose models: Wraps around popular ML libraries like 🤗 Transformers, allowing fast deployment of models through scalable server endpoints.
- Deploy detection models with tracking: Supports deployment of detection models from 🤗 Transformers (RT-DETR, DETR, Grounding DINO, etc.) with optional tracking integration.
- Aggregate robot-specific models from the robotics community: Intended as a platform for community-contributed multimodal models, usable in planning and control, especially with ROS components. See EmbodiedAgents.
Models And Wrappers
| Model Class | Description | Default Checkpoint / Resource | Key Init Parameters |
|---|---|---|---|
TransformersLLM | General-purpose large language model (LLM) from 🤗 Transformers | Qwen/Qwen3-0.6B | name, checkpoint, quantization, source, init_timeout |
TransformersMLLM | Multimodal vision-language model (MLLM) from 🤗 Transformers, with image and video inputs | Qwen/Qwen2.5-VL-3B-Instruct | name, checkpoint, quantization, source, init_timeout |
RoboBrain2 | Embodied planning + multimodal reasoning via RoboBrain 2.0 / 2.5 | BAAI/RoboBrain2.0-3B | name, checkpoint, source, init_timeout |
Whisper | Multilingual speech-to-text (ASR) from OpenAI Whisper | small.en (checkpoint list) | name, checkpoint, compute_type, source, init_timeout |
TransformersTTS | Text-to-speech via 🤗 Transformers (Bark, VITS, SpeechT5, SeamlessM4T, etc.) | suno/bark-small | name, checkpoint, voice, vocoder_checkpoint, source, init_timeout |
VisionModel | Detection + tracking via 🤗 Transformers | PekingU/rtdetr_r50vd_coco_o365 | name, checkpoint, setup_trackers, tracking_distance_threshold, num_trackers, source, init_timeout |
Installation
RoboML has been tested on Ubuntu 20.04 and later. A GPU with CUDA 12.1+ is recommended. If you encounter problems, please open an issue.
pip install roboml
From Source
git clone https://github.com/automatika-robotics/roboml.git && cd roboml
virtualenv venv && source venv/bin/activate
pip install pip-tools
pip install .
Model Checkpoints from ModelScope
By default, RoboML downloads model checkpoints from the HuggingFace Hub. However, users can pull checkpoints from ModelScope (魔搭社区) instead. First install the optional dependency (the Docker images below already include it):
pip install "roboml[modelscope]"
Then either set the source globally when starting the server:
ROBOML_SOURCE=modelscope roboml
# or with Docker: docker run -e ROBOML_SOURCE=modelscope ...
or per model, by passing source: "modelscope" in the body of the model's /initialize call. An explicit source parameter always overrides the environment variable.
ModelScope checkpoints are cached in ~/.cache/modelscope (configurable via MODELSCOPE_CACHE), so the cache mount shown in the Docker section covers them as well.
Most default checkpoints are available on ModelScope under the same IDs they have on the HuggingFace Hub, and Whisper size aliases such as small.en are resolved automatically. If a checkpoint cannot be found on ModelScope, model initialization fails with a descriptive error that suggests a suitable alternative checkpoint whenever one is known.
Gated and restricted checkpoints are detected automatically at initialization, on both hubs. Credentials are only required when a checkpoint actually needs them — in that case initialization fails with instructions for the relevant token: HF_TOKEN for HuggingFace Hub (or huggingface-cli login), or MODELSCOPE_API_TOKEN for ModelScope (or modelscope login).
Models that only exist on HuggingFace Hub can still be reached by routing HF downloads through a mirror, e.g. HF_ENDPOINT=https://hf-mirror.com (a community-run proxy; must be set before the server starts). This works independently of ROBOML_SOURCE, since every model node can choose its own source.
Vision Model Support
VisionModel uses HuggingFace Transformers for object detection and tracking. It works out of the box with any HuggingFace object detection model (RT-DETR, DETR, Grounding DINO, YOLOS, etc.). Object tracking via ByteTrack is included.
If ffmpeg or libGL is missing:
sudo apt-get update && apt-get install ffmpeg libsm6 libxext6
Docker Build (Recommended)
Jetson users are especially encouraged to use Docker.
- Install Docker Desktop
- Install the NVIDIA Container Toolkit
git clone https://github.com/automatika-robotics/roboml.git && cd roboml
# Build container image
docker build --tag=automatika:roboml .
# For Jetson boards:
docker build --tag=automatika:roboml -f Dockerfile.Jetson .
# Run HTTP server
docker run --runtime=nvidia --gpus all --rm -p 8000:8000 automatika:roboml roboml
# Or run RESP server
docker run --runtime=nvidia --gpus all --rm -p 6379:6379 automatika:roboml roboml-resp
-
(Optional) Mount your cache dir to persist downloaded models:
-v ~/.cache:/root/.cache
Servers
RoboML uses Ray Serve to host models as scalable apps across various environments.
WebSocket Endpoint
WebSocket endpoints are exposed for streaming use cases (e.g., STT/TTS).
Experimental RESP Server
For ultra-low latency in robotics, RoboML also includes a RESP-based server compatible with any Redis client.
RESP (see spec) is a lightweight, binary-safe protocol. Combined with msgpack instead of JSON, it enables very fast I/O, ideal for binary data like images, audio, or video.
This work is inspired by @hansonkd’s Tino project.
Usage
Run the HTTP server:
roboml
Run the RESP server:
roboml-resp
Example usage in ROS clients is documented in ROS Agents.
Running Tests
Install dev dependencies:
pip install ".[dev]"
Run tests from the project root:
python -m pytest
Copyright
Unless otherwise specified, all code is © 2024 Automatika Robotics. RoboML is released under the MIT License. See LICENSE for details.
Contributions
ROS Agents is developed in collaboration between Automatika Robotics and Inria. Community contributions are welcome!
Similar repos
Languages
Python
99.7%
RoboML is an aggregator package written for prototyping and deploying open source ML models for robotics
11
stars
137
commits
Python
primary language
Sep 3, 2026
updated
README
RoboML is an aggregator package for quickly deploying open-source ML models for robots. It supports three main use cases:
- Rapid deployment of general-purpose models: Wraps around popular ML libraries like 🤗 Transformers, allowing fast deployment of models through scalable server endpoints.
- Deploy detection models with tracking: Supports deployment of detection models from 🤗 Transformers (RT-DETR, DETR, Grounding DINO, etc.) with optional tracking integration.
- Aggregate robot-specific models from the robotics community: Intended as a platform for community-contributed multimodal models, usable in planning and control, especially with ROS components. See EmbodiedAgents.
Models And Wrappers
| Model Class | Description | Default Checkpoint / Resource | Key Init Parameters |
|---|---|---|---|
TransformersLLM | General-purpose large language model (LLM) from 🤗 Transformers | Qwen/Qwen3-0.6B | name, checkpoint, quantization, source, init_timeout |
TransformersMLLM | Multimodal vision-language model (MLLM) from 🤗 Transformers, with image and video inputs | Qwen/Qwen2.5-VL-3B-Instruct | name, checkpoint, quantization, source, init_timeout |
RoboBrain2 | Embodied planning + multimodal reasoning via RoboBrain 2.0 / 2.5 | BAAI/RoboBrain2.0-3B | name, checkpoint, source, init_timeout |
Whisper | Multilingual speech-to-text (ASR) from OpenAI Whisper | small.en (checkpoint list) | name, checkpoint, compute_type, source, init_timeout |
TransformersTTS | Text-to-speech via 🤗 Transformers (Bark, VITS, SpeechT5, SeamlessM4T, etc.) | suno/bark-small | name, checkpoint, voice, vocoder_checkpoint, source, init_timeout |
VisionModel | Detection + tracking via 🤗 Transformers | PekingU/rtdetr_r50vd_coco_o365 | name, checkpoint, setup_trackers, tracking_distance_threshold, num_trackers, source, init_timeout |
Installation
RoboML has been tested on Ubuntu 20.04 and later. A GPU with CUDA 12.1+ is recommended. If you encounter problems, please open an issue.
pip install roboml
From Source
git clone https://github.com/automatika-robotics/roboml.git && cd roboml
virtualenv venv && source venv/bin/activate
pip install pip-tools
pip install .
Model Checkpoints from ModelScope
By default, RoboML downloads model checkpoints from the HuggingFace Hub. However, users can pull checkpoints from ModelScope (魔搭社区) instead. First install the optional dependency (the Docker images below already include it):
pip install "roboml[modelscope]"
Then either set the source globally when starting the server:
ROBOML_SOURCE=modelscope roboml
# or with Docker: docker run -e ROBOML_SOURCE=modelscope ...
or per model, by passing source: "modelscope" in the body of the model's /initialize call. An explicit source parameter always overrides the environment variable.
ModelScope checkpoints are cached in ~/.cache/modelscope (configurable via MODELSCOPE_CACHE), so the cache mount shown in the Docker section covers them as well.
Most default checkpoints are available on ModelScope under the same IDs they have on the HuggingFace Hub, and Whisper size aliases such as small.en are resolved automatically. If a checkpoint cannot be found on ModelScope, model initialization fails with a descriptive error that suggests a suitable alternative checkpoint whenever one is known.
Gated and restricted checkpoints are detected automatically at initialization, on both hubs. Credentials are only required when a checkpoint actually needs them — in that case initialization fails with instructions for the relevant token: HF_TOKEN for HuggingFace Hub (or huggingface-cli login), or MODELSCOPE_API_TOKEN for ModelScope (or modelscope login).
Models that only exist on HuggingFace Hub can still be reached by routing HF downloads through a mirror, e.g. HF_ENDPOINT=https://hf-mirror.com (a community-run proxy; must be set before the server starts). This works independently of ROBOML_SOURCE, since every model node can choose its own source.
Vision Model Support
VisionModel uses HuggingFace Transformers for object detection and tracking. It works out of the box with any HuggingFace object detection model (RT-DETR, DETR, Grounding DINO, YOLOS, etc.). Object tracking via ByteTrack is included.
If ffmpeg or libGL is missing:
sudo apt-get update && apt-get install ffmpeg libsm6 libxext6
Docker Build (Recommended)
Jetson users are especially encouraged to use Docker.
- Install Docker Desktop
- Install the NVIDIA Container Toolkit
git clone https://github.com/automatika-robotics/roboml.git && cd roboml
# Build container image
docker build --tag=automatika:roboml .
# For Jetson boards:
docker build --tag=automatika:roboml -f Dockerfile.Jetson .
# Run HTTP server
docker run --runtime=nvidia --gpus all --rm -p 8000:8000 automatika:roboml roboml
# Or run RESP server
docker run --runtime=nvidia --gpus all --rm -p 6379:6379 automatika:roboml roboml-resp
-
(Optional) Mount your cache dir to persist downloaded models:
-v ~/.cache:/root/.cache
Servers
RoboML uses Ray Serve to host models as scalable apps across various environments.
WebSocket Endpoint
WebSocket endpoints are exposed for streaming use cases (e.g., STT/TTS).
Experimental RESP Server
For ultra-low latency in robotics, RoboML also includes a RESP-based server compatible with any Redis client.
RESP (see spec) is a lightweight, binary-safe protocol. Combined with msgpack instead of JSON, it enables very fast I/O, ideal for binary data like images, audio, or video.
This work is inspired by @hansonkd’s Tino project.
Usage
Run the HTTP server:
roboml
Run the RESP server:
roboml-resp
Example usage in ROS clients is documented in ROS Agents.
Running Tests
Install dev dependencies:
pip install ".[dev]"
Run tests from the project root:
python -m pytest
Copyright
Unless otherwise specified, all code is © 2024 Automatika Robotics. RoboML is released under the MIT License. See LICENSE for details.
Contributions
ROS Agents is developed in collaboration between Automatika Robotics and Inria. Community contributions are welcome!
Similar repos
Languages
Python
99.7%