OpenForge RL

Train harness-native agents in any environment.

Xiao Yu, Baolin Peng, Ruize Xu, Hao Zou, Qianhui Wu, Hao Cheng, Wenlin Yao, Nikhil Singh, Zhou Yu, Jianfeng Gao · Columbia University · Dartmouth College · Microsoft Research · Paper

⚠️ Disclaimer: This repository is still being actively reorganized/debugged — expect rough edges, in-flight renames, and scripts that reference paths or checkpoints that are not yet public. However, most of the main scripts and logics should be functional so feel free to experiment/try out the code! We are actively updating this codebase in the coming weeks, please stay tuned for more updates and improvements. If you have any questions or issues, please reach out to the authors or open an issue on GitHub.

Roadmap:

• Initial release
• Upload demo task and environment data
• Test and integrate with new veRL release to support Qwen3.5+ training
• Upload full training tasks and data to huggingface datasets

---

Motivation

An agent in production is not just a model — it is a model plus a harness: the scaffold that builds prompts, parses tool calls, manages history, and retries. But agents are usually trained in a simplified stand-in for that harness, so the policy you train is not the policy you ship.

OpenForge RL trains the agent inside the real harness, against the real environment. A rollout is a full episode: the actual harness drives the actual environment, and every model call along the way becomes training data.

The payoff is combinatorial — any harness × any environment, with standard RL codebases (veRL) underneath:

How it works

1. Orchestrator + sandboxes — every rollout runs in its own container, reachable over HTTP, so rollouts scale out while training stays on the GPU host.
2. A recording proxy — the harness calls what looks like a normal OpenAI-compatible endpoint; the proxy serves the request and records every prompt/response pair. This is what makes an unmodified harness trainable.
3. The trainer — veRL consumes the recorded episodes for multi-turn RL (GRPO / MM-GRPO / DAPO, Megatron or FSDP).

Environments (openforgeenvs/)

One of the repo's two core pieces — the other is openforgerl/, the rollout engine that adapts rollouts running in remote containers to training engines like veRL. Everything under openforgeenvs/ is inference/run logic — the code that takes a task, sets up a sandbox, drives the episode, and scores it. The tasks themselves are data and live elsewhere (see below).

The rollout engine supports several modes of agent↔environment interfacing (see openforgerl/rollout_core.py), so developers choose how much of the loop they own:

PipelineEnvInstanceConfig — blackbox pipeline (recommended, most flexible). The env owns the entire episode: the engine hands it a fresh sandbox plus an env_run_config (including llm_args), and the env sets up the pod, runs the agent harness (e.g. the real codex CLI), verifies, and returns the reward. Implementing one takes a single env.py:

# openforgeenvs/<name>/env.py — all a Pipeline env needs
def create_env(sandbox):
    return MyEnv(sandbox)

class MyEnv:
    async def run(self, **env_run_config) -> EnvCallResult:
        ...  # set up the pod, run the harness against llm_args,
             # verify, and return {"reward": ..., ...}

Claw and computer use (claw_eval/, osworld/) are Pipeline envs. Because the harness only sees an OpenAI-compatible endpoint, pointing llm_args at the recording proxy captures training data with zero harness changes.

DirectEnvInstanceConfig — engine-driven agent loop. The engine's own ReACT loop steps the env turn by turn; the env wraps the sandbox with step-style methods, and prompts.py/projection.py map model text to env actions. The browser env (onlinem2w-molmo) works this way — its fuller layout adds env_server.py (in-pod HTTP API), projection.py, prompts.py, and a Dockerfile/build_push.sh for the env image (details in openforgeenvs/README.md).

The tasks themselves are data, not repo code. Training uses synthetic tasks built by our synthesis pipeline in all three domains (to be released separately, e.g. as HuggingFace datasets). A claw task, for instance, is a folder — task.yaml (instruction), environment/ (fixtures + mock services), tests/ (the verifier) — paired with a prebuilt task image.

Shipped environment runtimes: claw_eval (CLI/tool agents; evaluated on ClawEval), osworld (computer use; evaluated on OSWorld-Verified), onlinem2w-molmo (browser use; evaluated on Online-Mind2Web / WebVoyager).

Data

Every task needs a verifier, so tasks are built, not scraped — by a task synthesis pipeline (propose → prune → build → test → refine; see the paper). The pipeline will be released separately as its own repo.

In-repo, tasks flow through two parquet stages: a raw parquet (task fields) and a rollout parquet that adds the orchestrator URL, environment image, judge configuration, and step limits. The rollout parquet is the single unit consumed by both training and inference — which is why they share one code path. The scripts below call the converters for you.

Installation

git clone https://github.com/MSR-Orchard/OpenForge-RL.git
cd OpenForge-RL
uv sync

third_party/verl

verl is a git submodule (empty after a plain clone) and needs our patch on top of the pinned commit:

git submodule update --init third_party/verl
USE_MEGATRON=1 USE_SGLANG=0 scripts/install_vllm_sglang_mcore.sh
cd third_party/verl
git apply ../verl.patch   # multi-turn rollout hooks, reward plumbing, proxy port
uv pip install -e .
uv pip uninstall nvidia-cudnn-cu12

The patch lives in the submodule's working tree and is not committed — any git submodule update or git checkout inside third_party/verl wipes it; re-apply with git apply ../verl.patch.

orchard_env (sandbox orchestration)

Sandbox orchestration comes from Orchard (orchard_env). It is not on PyPI, but uv sync already installs it automatically as a git dependency — no manual step needed. Only if you want to hack on Orchard itself, override with a local editable install:

git clone https://github.com/microsoft/Orchard.git
uv pip install -e "Orchard/orchard_env[dev]"

Quick start

Shared setup — point the scripts at your sandbox orchestrator (these are the env vars orchard_env's client reads):

export SANDBOX_BASE_URL="http://<orchestrator-host>:80"
export SANDBOX_API_KEY="<key>"

Claw (CLI agents: Codex / OpenClaw / ZeroClaw)

# 1. host the policy model
CUDA_VISIBLE_DEVICES=0,1 bash examples/vllm/deploy_openforge-claw.sh

# 2. inference / eval on ClawEval — one runner per harness
bash examples/inference/run_claweval_codex_e2e.sh      # also: openclaw, zeroclaw

# 3. RL training (GRPO, Megatron)
bash examples/verl/train_claw_grpo_qwen30ba3ab-megatron.sh

Claw scripts are configured in the block at the top of each file (model path/id, task parquet, engine URL). Rollouts route the harness through the recording proxy, which the sandboxes must be able to reach — set OPFORGE_VLLM_PROXY_PUBLIC_IP / OPFORGE_VLLM_PROXY_PUBLIC_PORT. No public ports on your GPU host? See the relay recipes in examples/verl/README.md.

GUI — computer use

# 1. host the policy model
CUDA_VISIBLE_DEVICES=0 bash examples/vllm/deploy_openforge-computeruse.sh

# 2. RL training (MM-GRPO, Megatron); configured in the script header like claw
bash examples/verl/train_computeruse_grpo_qwen3vl8b-megatron.sh

For OSWorld-Verified evaluation, point the official OSWorld harness at the endpoint deployed in step 1.

GUI — browser use

# 1. host the policy model
CUDA_VISIBLE_DEVICES=0 bash examples/vllm/deploy_openforge-browser.sh

# 2. inference + in-env eval — the run's own success_rate IS the benchmark score
bash examples/inference/run_browser_om2w.sh

# 3. RL training (MM-GRPO)
bash examples/verl/train_browser_grpo_qwen3vl8b-megatron.sh

Like claw, browser scripts are configured in the block at the top of each file (endpoints, env image, judge, n_samples/n_parallel). The browser agent loop runs client-side, so no recording proxy is involved. run_eval_browser.sh re-judges an existing run, or scores WebVoyager. More detail on all three domains: examples/{vllm,inference,verl}/README.md.

Repo layout

openforgerl/        core: rollout engine — agent loop, recording proxy, and the
                    glue adapting remote-container rollouts to veRL
openforgeenvs/      core: environment implementations
examples/verl/      RL training scripts
examples/inference/ inference + offline scoring
examples/vllm/      model hosting
third_party/verl    veRL submodule (patched, see above)

Sandbox orchestration (the Kubernetes orchestrator and its client) lives in orchard_env; this repo's OpenForge extensions over its client are in openforgerl/orchard_compat.py.

Citation

@misc{yu2026openforgerltrainharnessnativeagents,
      title={OpenForgeRL: Train Harness-native Agents in Any Environment}, 
      author={Xiao Yu and Baolin Peng and Ruize Xu and Hao Zou and Qianhui Wu and Hao Cheng and Wenlin Yao and Nikhil Singh and Zhou Yu and Jianfeng Gao},
      year={2026},
      eprint={2607.21557},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2607.21557}, 
}