Dressage: Scalable RL for Any Agent and Sandbox

Dressage is an agentic reinforcement learning training framework built on top of slime. It bridges the gap between policy rollouts, sandboxed tool execution, and training data conversion through a shared proxy and paddock layer.

Dressage lets you train diverse types of LLM agents that use real tools — like code editors, shell commands, file I/O, retrieval APIs — with full RL gradient flow. Both whitebox (Python tool loops) and blackbox (HTTP agents like opencode, openclaw, claude_code, and codex) paradigms are supported through a unified interface.

Table of Contents

• News
• Technical Highlights
• Architecture Overview
• Quick Start
• Project Structure
• Documentation
• Docker
• Contributing
• Acknowledgements
• License

📢 News

• [2026/08/27] ⚖️ Rollout Step Balance Released. From Session Sticky to Step Balance, it mitigates Engine load imbalance caused by long-tail sessions. Compared with the sticky baseline, 8-Engine Qwen3.5-4B experiments across different workload distributions showed 39.4%–64.2% higher Effective TPS/GPU and 28.2%–38.9% shorter rollout time. Report: English | 中文
• [2026/08/14] 💾 Integrated TransferQueue for scalable trajectory storage: Offloads R3 routed_experts and logprobs into independent StorageUnits, reducing peak master-node trajectory data-plane memory by 67% in a two-node Qwen3.6 experiment. In the 32-node GLM-5.2 capacity analysis, the reduction reaches 91%. Report: English | 中文
• [2026/07/26] 💻 Dressage SWE-Gym blackbox RL recipe released. The recipe provides an end-to-end Claude Code blackbox RL pipeline with fresh-sandbox evaluation, anti-cheating safeguards, and a reproducible 64K setup. Qwen3.5-4B trained on SWE-Gym improves accuracy on SWE-bench Verified from 32.6% to 37.8% (+5.2 points). Report: English | 中文 | Training script
• [2026/07/25] 🔥 Dressage Claw blackbox RL recipe released. The recipe provides an end-to-end OpenClaw blackbox RL pipeline with staged grader injection and a reproducible 64K setup. Qwen3.6-35B-A3B trained on Dressage-Claw improves strict Pass^3 on the ClawEval general split from 41.4 to 55.4 (+14.0 points). Report: English | 中文 | Training script
• [2026/07/23] Support for Multi-Teacher OPD (MOPD): Train a single student across any agent with multiple frozen teachers, routing each dataset to its assigned teacher while serially rotating teachers through shared GPU model buffers. Read the technical guide.
• [2026/07/19] Integrated Harbor for rollout and training: Harbor orchestrates task execution and evaluation, while Dressage handles model routing and trajectory collection.
• [2026/07/09] Multi-Agent Training. From Timeline to Lineage, Lineage-aware TITO builds correct multi-agent training trajectories by reconstructing context inheritance across main-agent and subagent calls. Read the technical deep dive.
• [2026/07/04] Multi-Harness Training. Supports Claude Code and Codex, and you can train multiple harnesses in a single script.
• [2026/06/30] Released Whitebox agent training curves and True Staleness Control.
• [2026/06/20] Dressage is now open source!

💡 Technical Highlights

Dressage introduces several key innovations for agentic RL:

Any Agent, Any Sandbox

Dressage separates agent semantics from execution placement. Whitebox Python tool loops, blackbox HTTP agents such as opencode, openclaw, claude_code, and codex, local bubblewrap pools, and remote E2B sandboxes all converge on the same proxy-to-training path.

Token-Wise Control

Dressage records training evidence at token granularity: token_id, logprob, loss_mask, token_version, and token_expert. TITO — Token-In-Token-Out avoids retokenization drift by encoding only each turn's append delta and splicing token IDs incrementally.

The same token-wise model powers Token-Boundary Pause/Resume through GenerationController, version-aware masking through token_version, and MoE Routing Replay (R3) through token_expert. See Training Layer for the deep dive.

Segment-Aware Training

When trajectories split because of history compaction or tool-schema changes, Dressage expands every segment into a training sample and broadcasts the anchor segment's terminal advantage to its siblings. Shared rollout_id / parent_traj_id values keep segments in the same slime training step, while prompt-equal denominators prevent split-heavy trajectories from receiving extra gradient weight, please refer to Training Layer.

Slime-Native Integration

Dressage is built on slime, not maintained as a fork. Rollout generation, reward processing, sample conversion, and training behavior plug into upstream slime through dotted import-path hooks.

Production-Grade Rollout Safety

Atomic trajectory logging, HTTP error redaction, context overflow detection, session artifact archiving, and abort safety contracts keep failed rollouts diagnosable without corrupting training data.

🏗️ Architecture Overview

Dressage's architecture is built on an elegant separation of concerns across three orthogonal axes. The entire system sits on top of slime as a git submodule, extending it through dotted import-path customization hooks — no upstream fork required.

Compact mental model:

• Dressage owns the agentic RL bridge: rollout hooks create sessions, the proxy records token-level evidence, and the training layer converts trajectories into slime-ready samples.
• Paddocks own interaction semantics: whitebox paddocks run Python tool loops; blackbox paddocks delegate to HTTP agents through BlackboxServer.
• Sandboxes own placement and isolation: the same rollout path can target local bubblewrap slots or remote E2B sandboxes without changing the agent logic.
• Slime owns the training substrate: Megatron training, Ray rollout orchestration, and SGLang inference stay upstream and are customized through import-path hooks.

The key insight: paddock mode (whitebox vs blackbox) and sandbox provider (local vs remote) are orthogonal axes — mix and match freely. The sandbox layer now exposes provider-neutral SandboxSpec, SandboxLease, and service endpoints; paddocks create and terminate leases while blackbox rollouts resolve the blackbox service endpoint from the lease.

Both whitebox and blackbox agent scaffolds converge on the same post-rollout path:

proxy.finalize_session → trajectory/read → expand_segments_to_samples → list[Sample]

End-to-End Rollout Flow

1️⃣  Generate hook creates a paddock lease + proxy session
2️⃣  Agent runs (whitebox Python loop or blackbox HTTP agent)
3️⃣  Each LLM call goes through Proxy → SGLang; tokens recorded per step
4️⃣  Session finalizes into one or more trajectory segments
5️⃣  Segments expand to training Samples; reward + loss hooks prepare train_data

🚀 Quick Start

Start Docker Environment

docker pull huang3eng/dressage:latest

docker run --rm --gpus all --ipc=host --shm-size=16g \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -it huang3eng/dressage:latest /bin/bash

For reproducible environments, pin a dated tag such as huang3eng/dressage:nightly-dev-20260704a.

Prepare Model Checkpoint

hf download Qwen/Qwen3.5-4B \
  --local-dir /root/Qwen3.5-4B

cd /root/Dressage
git submodule update --init --recursive
git submodule status --recursive
cd slime
source scripts/models/qwen3.5-4B.sh

PYTHONPATH=/root/Megatron-LM python tools/convert_hf_to_torch_dist.py \
  "${MODEL_ARGS[@]}" \
  --hf-checkpoint /root/Qwen3.5-4B \
  --save /root/Qwen3.5-4B_torch_dist

Run a Training Example

cd /root/Dressage
bash examples/scripts/run_blackbox_qwen3.5_4b_async_local.sh

For detailed setup instructions, all configuration options, and troubleshooting, see the Quick Start Guide. To prepare and run the bundled ALFWorld and HotpotQA whitebox recipes, see the Whitebox Agent Quick Start.

📁 Project Structure

dressage/
├── proxy/                 # Inference proxy, trajectory recording, TITO
│   └── tito/              # Token-In-Token-Out incremental tokenizer
├── paddock/               # Environment interaction (blackbox | whitebox)
├── sandbox/               # Isolation backends (bwrap, E2B)
│   ├── local/bwrap/       # Ray-managed bubblewrap pool
│   └── remote/e2b/        # E2B provider
├── rollout/               # Generate hooks, multi-segment, async rollout
│   └── generate/          # Blackbox dispatch + whitebox agent scaffolds
├── training/              # Reward post-process, async train with pause
├── reward/                # Sample-oriented reward registry
├── recipes/               # Example agents (alfworld, hotpotqa)
└── config/                # Shared defaults and env vars

blackbox_server/
├── api/                   # FastAPI route handlers
├── adapters/              # Backend adapters (opencode, openclaw, claude_code, codex)
├── core/                  # Server logic, models, monitoring
├── proxy/                 # In-process LLM proxy with session headers
├── store/                 # In-memory session store
└── runtime/               # Path and runtime ID resolution

slime/                     # Git submodule — upstream RL framework (v0.3.0)
examples/
├── scripts/               # Training run scripts (sync, async, partial)
└── data/                  # Sample/demo prompt datasets (DAPO Math)
tests/                     # 30+ test modules
docs/                      # Protocol and implementation docs
docker/                    # Reproducible local environment

Core modules map onto the main runtime layers:

Inference Proxy: Token-Level Trajectory Recording

OpenAI-compatible HTTP service that sits between agent rollouts and the SGLang inference router. Every LLM call passes through it — every token, logprob, and loss mask is captured for training. Agents never call SGLang directly. Supports per-step recording, auto-segmentation on history rewrites, preemptible generation via GenerationController, weight version tracking, and MoE routing replay. The central nervous system of trajectory recording.

Paddock: Unified Environment Interaction

Single-class abstraction that manages all environment interaction during rollouts — sandbox lease creation, agent/tool calls, pause/resume lifecycle, and cleanup. The "what-to-do" layer. BlackboxPaddock drives HTTP agents (register_agent → call_agent → pause/resume), while WhiteboxPaddock exposes tool_call for Python agents (shell, file read/write). Factory pattern: create_paddock_from_env() wires everything from environment variables.

Sandbox Backends: Pluggable Isolation

Two pluggable isolation backends, swapped via a single environment variable. Local bubblewrap (local_bwrap): Ray-managed pool of bubblewrap-isolated slots with supervisor health monitoring. E2B (e2b): elastic cloud-native sandboxes from E2B templates/images. Two local pool modes: blackbox (full BlackboxServer service endpoint) or command_only (shell/file ops). Local blackbox clusters can also be managed with the dressage-local-blackbox-* / dressage-blackbox-* scripts.

BlackboxServer: Unified HTTP Adapter

Bundled HTTP adapter service that decouples the rollout manager from concrete agentic backends. Sits inside sandboxes, manages exactly one backend agent process and one active session at a time, and transparently proxies all LLM calls back through the Dressage inference proxy. Supports opencode, openclaw, claude_code, and codex through a pluggable adapter pattern. Features turn idempotency, register-and-rebind, and background health monitoring.

Rollout Hooks: Slime Integration

Bridges slime's RL training loop with Dressage's agentic capabilities through customizable hooks — all specified as dotted import paths, no fork required. Two generate paradigms: whitebox_agent (Python tool loops) and blackbox_dispatch (BlackboxServer delegation). Three async scheduling modes: sync, fully async (background worker pipelines), and partial async (early return on global_batch_size). Includes pluggable reward registry, prompt-equal sample conversion, and trajectory-level logging.

Training Layer: Multi-Segment & TITO

Transforms proxy-recorded trajectories into slime-compatible training data. Multi-segment expansion trains on all segments (not just the last). TITO tokenization eliminates retokenization drift across turns. Prompt-equal gradient aggregation ensures fair scaling for GRPO. Partial rollout resume preserves in-flight generation across weight updates. Reward post-processing broadcasts anchor advantages to sibling segments.

Recipes: Agent Training Recipes

End-to-end whitebox and blackbox RL recipes for SWE-Gym, Dressage Claw, ALFWorld, and HotpotQA, with data preparation, training scripts, reward and evaluation implementations, and a build-your-own guide.

📚 Documentation

• Proxy — Inference proxy architecture, session model, trajectory build modes, routing replay
• Paddock — Paddock interface, blackbox vs whitebox interaction, factory pattern
• Sandbox — Sandbox providers, bwrap pools, E2B integration
• BlackboxServer — HTTP adapter protocol, backends, in-process LLM proxy, session states
• Rollout — Generate hooks, async modes, reward registry, slime wiring
• Training — Multi-segment, TITO, prompt-equal aggregation, partial rollout
• Harbor Integration — Harbor rollouts, trainable trajectory capture, and synchronous slime training
• Quick Start — Step-by-step setup, configuration reference, troubleshooting
• Recipes — End-to-end whitebox and blackbox training recipes for SWE-Gym, Dressage Claw, ALFWorld, and HotpotQA, with quickstarts, experiment reports, and a build-your-own guide.
  • SWE-Gym + Claude Code — Quickstart (中文) and experiment report (中文)
  • Dressage Claw + OpenClaw — Experiment report (中文)
  • Whitebox Agents — Quick Start with data preparation and launch commands for ALFWorld and HotpotQA

🐳 Docker

A reproducible local environment is provided via Docker:

docker/build.sh    # Build the image
docker/run.sh      # Run with --gpus all --network host --ipc host --privileged

The image includes bubblewrap, opencode, openclaw, Claude Code, Codex CLI, Dressage, and BlackboxServer. --privileged is required for bubblewrap inside containers. See docker/README.md for details.

👥 Authors and Contributors

Dressage is developed and maintained by its authors and contributors.

Core Contributors (alphabetical): Liangmeng Huang (huang3eng@gmail.com) Hao Dong (dh20373967@gmail.com) Qingchuan Li (lqcustc@gmail.com) Hongwei Xue (xuehongwe@gmail.com) Shilin Yan (tattoo.ysl@gmail.com) For academic collaborations, citations, or technical inquiries, please contact us.

Contributors (alphabetical): Wenhui Chen, Xueyuan Han, Jing He, Hongyu Li, Junbo Li, Zicheng Liu, Sibo Wei, Senyu Zhang, Guannan Zhang.

To cite Dressage:

@misc{dressage_github,
  author       = {Liangmeng Huang and Qingchuan Li and Hongwei Xue and Shilin Yan and {Dressage Contributors}},
  title        = {{Dressage}: Scalable {RL} for Any Agent and Any Sandbox},
  year         = {2026},
  howpublished = {\url{https://github.com/Accio-Lab/Dressage}}
}

🤝 Contributing

Contributions are welcome! If you have suggestions for new features, performance tuning, or feedback on user experience, feel free to submit an Issue or PR.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Ensure tests pass (pytest)
4. Commit your changes (git commit -m 'Add amazing feature')
5. Push to the branch (git push origin feature/amazing-feature)
6. Open a Pull Request

Development notes:

• When bumping the slime submodule, diff convert_samples.py against upstream slime/ray/rollout.py
• TITO currently supports qwen3_5 and qwen3_6; contributions for additional model templates are welcome.
• The claude_code adapter runs Claude Code headless through the BlackboxServer Anthropic Messages bridge.
• The codex adapter runs codex exec --json with an isolated sandbox-local CODEX_HOME, points Codex at the in-process Dressage proxy, and does not mount host ~/.codex credentials.

🙏 Acknowledgements

We gratefully acknowledge:

• slime (THUDM, Tsinghua University) — The upstream RL post-training framework that Dressage builds upon. slime provides the foundational Megatron training loop, Ray rollout management, and SGLang inference integration.
• SGLang — High-performance inference engine powering the generation backend.
• Megatron-LM (NVIDIA) — Distributed training infrastructure used via slime.
• bubblewrap — Unprivileged sandboxing for local agent isolation.

📄 License

This project is licensed under the Apache License 2.0 — see the LICENSE file for details.

The Docker setup can install opencode, openclaw, Claude Code, and Codex CLI using their public install scripts. Those tools are not source code distributed by this repository; review their upstream licenses before redistribution.