HuggingEnvs — RL Environments 101: building and scaling RL environments in the age of LLMs
206
stars
96
commits
Python
primary language
Sep 9, 2026
updated
README
HuggingEnvs
Open source RL environments for LLM agents
Build · Deploy · Train · Eval — end to end, reproducible, in the open.
Why this exists
The RL environment ecosystem is moving fast. New frameworks land every few weeks, each with its own vocabulary for the same handful of ideas, and most of what's written about them is either a launch post or a spec. Meanwhile the actual bottleneck in RL for LLMs has quietly shifted: the algorithm isn't the hard part any more — the environment is.
So we're building the resource we wanted. Open source, end to end, and reproducible: how to design an environment, how to build it, how to deploy it, how to train against it, and how to scale it to thousands of concurrent sessions. Real code you can run, not diagrams of code someone else ran.
Every environment here works. Every rollout has been executed. Every training curve came from a job you can launch yourself, in one command, without a GPU of your own.
4 environments · 6 frameworks · 19 implementations · 10 deployed Spaces · 5 agent skills
Where things live: source, recipes and notebooks in this repo · environments, datasets, models and demos on 🤗 huggingface.co/HuggingEnvs.
Projects
Each numbered folder is a self-contained, end-to-end project — its own environments, notebooks, results and README, plus the Hub repos it owns. They read in order but stand alone.
| # | Project | What you get | Envs | Frameworks | Deployed | Status |
|---|---|---|---|---|---|---|
| 00 | RL Environments 101 | Three environments, six frameworks, side by side. | 3 | 6 | 8 | ✅ stable |
| 01 | LaTeX OCR | Train Qwen3-VL-2B to read math images into LaTeX, with a verifiable reward. | 1 | 1 | 1 | 📓 notebook |
| 02 | Watercolour | Train Qwen3.5-35B-A3B to paint watercolours by writing p5.brush sketches, rewarded by an aesthetic preference model. | 1 | 1 | 0 | ✅ trained |
| 03 | GeoGuesser | Drop a VLM at a random street corner on Earth and score it on kilometres of error. | 1 | 1 | 1 | ✅ stable |
Generated from each project's project.yaml by tools/build_index.py. Adding a project means
editing that manifest, not this table.
00 · RL Environments 101 3 environments · 6 frameworks · 8 live Spaces
One env, six ways. Three environments, each implemented six times — same logic, six framework
dialects. Diff any two server.py files and the differences are the lesson.
| Environment | Turns | Tools | Backend |
|---|---|---|---|
| Jupyter agent | multi | 4 | E2B sandbox, real code execution |
| Wordle | multi | 1 | pure Python, no backend |
| Desktop | multi | 19 | E2B Desktop, vision-driven |
Across openenv · ors · nemo_gym · verifiers · skyrl_gym · gem — 18 implementations, 8 of
them deployed as Spaces you can hit right now.
01 · LaTeX OCR train a VLM against a served reward
The full loop. Project 00 shows you what an environment is; this one takes a single environment all the way to a trained model.
Qwen3-VL-2B learns to read rendered math into LaTeX with GRPO, scored by a reward served from a live OpenEnv Space. Correctness is checkable — render the prediction, compare — so the reward is honest and there is very little to game. Runs on a GPU you spin up in one command; no cluster, no local GPU.
02 · Watercolour train against an aesthetic reward
RL over taste. Project 01 trains against a verifiable reward. This one asks what happens when there is no correct answer at all.
Qwen3.5-35B-A3B learns to paint watercolours by writing p5.brush sketches. Headless Chromium renders each sketch, and the render is scored by an aesthetic preference model plus a VLM judge comparing it against a hand-rated reference pool, so the reward is somebody's taste. Three reward mixes trained and compared, every artifact published, and the full story in the blog post.
03 · GeoGuesser train a VLM to place itself on Earth
A game as an environment. Projects 01 and 02 score one shot. This one is multi-turn: the agent looks around, walks down the road, zooms on a sign, pins a candidate, and commits.
Qwen3.5-4B learns visual geolocation with GRPO against a
playable OpenEnv Space serving real
Mapillary panoramas, scored on kilometres of error. It ends up ahead of gpt-5.4-mini and
claude-haiku-4.5 on a 200-task held-out split and behind only claude-sonnet-5, for about $100.
Ten hours on four A100s, or one GPU if you are in no hurry. Three training runs, the reward
redesigned once, and every measurement bug written down in
the article.
More coming. Each new project is another end-to-end recipe: an environment, a training run, and the artifacts on the Hub. Proposals and contributions welcome →
Articles & talks
Long-form writing and conference talks. Sources live in content/; each one ships to the
Hub as a Space.
|
How to turn a game into an RL environment
The technical intuition, worked end to end: curating the data, designing the environment, shipping it
with OpenEnv, and training a 4B against it with TRL until it outscored 📂 |
The Ultimate Guide to RL Environments
Building and scaling RL environments in the LLM era — how frameworks are built, how rewards are wired, and how they scale to thousands of concurrent sessions. |
RL Environments 101
From "what is an env?" to training your own. RL fundamentals → environment anatomy → OpenEnv → training with TRL. The original 30-minute talk. |
Scaling RL for LLMs
What an environment actually is, how reward hacking happens, and how to build and train against your own. The 20-minute cut, for AMD AI Dev Day. |
|
Multi-Harness Training
OpenEnv × Harbor — why an environment's failure model decides whether it can be trained against: in-process agent loops vs. an HTTP boundary, and what it takes to capture trainable tokens. |
Quickstart
Run an environment. Wordle is pure Python with no external backend — the fastest full rollout:
git clone https://github.com/adithya-s-k/HuggingEnvs
cd HuggingEnvs
cp .env.example .env # HF_TOKEN, plus E2B_API_KEY for sandbox-backed envs
cd 00-environments-101/envs/wordle/verifiers
uv sync && uv run python rollout.py
Train a model against one. No GPU, no cluster, no setup — one command spins up a GPU with the notebooks loaded and prints a JupyterLab URL:
curl -sSL https://raw.githubusercontent.com/adithya-s-k/HuggingEnvs/main/tools/jupyter_launch.py | python3 -
Windows (PowerShell): irm https://raw.githubusercontent.com/adithya-s-k/HuggingEnvs/main/tools/jupyter_launch.py | python -. Set FLAVOR=t4-small for a cheaper GPU. Track jobs at huggingface.co/settings/jobs.
Build your own environment
Five SKILL.md-spec agent skills turn a plain-English description into a runnable RL environment across four frameworks. They work in any project — with Claude Code, Cursor, Codex, OpenCode, Gemini CLI and others.
npx skills add adithya-s-k/HuggingEnvs
| Skill | What it builds |
|---|---|
rl-env-from-description | Orchestrator — interviews you, then ports across all four frameworks |
generate-openenv-env | OpenEnv (Hugging Face / Meta) — HTTP + MCP |
generate-ors-env | OpenReward (ORS) — per-tool-call rewards |
generate-verifiers-env | Verifiers (Prime Intellect) — in-process + rubrics |
generate-nemo-gym-env | NeMo Gym (NVIDIA) — HTTP + post-episode /verify |
"make me an env where the agent plays connect-four" — that's the whole interface.
Repository layout
HuggingEnvs/
├── 00-environments-101/ 3 environments × 6 frameworks
├── 01-latex-ocr/ train a VLM against a served reward
├── content/
│ ├── articles/ long-form sources (Astro → Docker Space)
│ └── slides/ talk decks (Vite → static Space)
├── tools/ launcher, Space deploy, index generation
├── assets/ brand + content thumbnails
└── .claude/skills/ the five environment-authoring agent skills
Inside a project the folders always mean the same thing: envs/ (implementations, shared logic in
core/), train/ (configs + launch), notebooks/, results/.
Contributing
We're actively looking for new end-to-end recipes — a task, an environment, a training run, and honest results. Domains we don't cover yet are especially welcome: web browsing, SQL, games, robotics sims, tool-use over real APIs, long-horizon software engineering.
Half-finished counts. A recipe with real numbers and a gap beats a polished one nobody ran — open an issue and we'll help you land it. New framework ports, reproductions that disagree with ours, and corrections to the guide are all just as welcome.
See CONTRIBUTING.md. The fastest path to a new environment is the
rl-env-from-description skill.
Citation
@misc{huggingenvs,
author = {Kolavi, Adithya S},
title = {HuggingEnvs: Open Source RL Environments for LLM Agents},
year = {2026},
url = {https://github.com/adithya-s-k/HuggingEnvs}
}
License
Contributors
81 commits
15 commits
Similar repos
Languages
Python
22.2%
JavaScript
21.9%
HTML
20.2%
TypeScript
12.7%
Astro
8.9%
TeX
6.7%
CSS
3.1%
MDX
2.5%
HuggingEnvs — RL Environments 101: building and scaling RL environments in the age of LLMs
206
stars
96
commits
Python
primary language
Sep 9, 2026
updated
README
HuggingEnvs
Open source RL environments for LLM agents
Build · Deploy · Train · Eval — end to end, reproducible, in the open.
Why this exists
The RL environment ecosystem is moving fast. New frameworks land every few weeks, each with its own vocabulary for the same handful of ideas, and most of what's written about them is either a launch post or a spec. Meanwhile the actual bottleneck in RL for LLMs has quietly shifted: the algorithm isn't the hard part any more — the environment is.
So we're building the resource we wanted. Open source, end to end, and reproducible: how to design an environment, how to build it, how to deploy it, how to train against it, and how to scale it to thousands of concurrent sessions. Real code you can run, not diagrams of code someone else ran.
Every environment here works. Every rollout has been executed. Every training curve came from a job you can launch yourself, in one command, without a GPU of your own.
4 environments · 6 frameworks · 19 implementations · 10 deployed Spaces · 5 agent skills
Where things live: source, recipes and notebooks in this repo · environments, datasets, models and demos on 🤗 huggingface.co/HuggingEnvs.
Projects
Each numbered folder is a self-contained, end-to-end project — its own environments, notebooks, results and README, plus the Hub repos it owns. They read in order but stand alone.
| # | Project | What you get | Envs | Frameworks | Deployed | Status |
|---|---|---|---|---|---|---|
| 00 | RL Environments 101 | Three environments, six frameworks, side by side. | 3 | 6 | 8 | ✅ stable |
| 01 | LaTeX OCR | Train Qwen3-VL-2B to read math images into LaTeX, with a verifiable reward. | 1 | 1 | 1 | 📓 notebook |
| 02 | Watercolour | Train Qwen3.5-35B-A3B to paint watercolours by writing p5.brush sketches, rewarded by an aesthetic preference model. | 1 | 1 | 0 | ✅ trained |
| 03 | GeoGuesser | Drop a VLM at a random street corner on Earth and score it on kilometres of error. | 1 | 1 | 1 | ✅ stable |
Generated from each project's project.yaml by tools/build_index.py. Adding a project means
editing that manifest, not this table.
00 · RL Environments 101 3 environments · 6 frameworks · 8 live Spaces
One env, six ways. Three environments, each implemented six times — same logic, six framework
dialects. Diff any two server.py files and the differences are the lesson.
| Environment | Turns | Tools | Backend |
|---|---|---|---|
| Jupyter agent | multi | 4 | E2B sandbox, real code execution |
| Wordle | multi | 1 | pure Python, no backend |
| Desktop | multi | 19 | E2B Desktop, vision-driven |
Across openenv · ors · nemo_gym · verifiers · skyrl_gym · gem — 18 implementations, 8 of
them deployed as Spaces you can hit right now.
01 · LaTeX OCR train a VLM against a served reward
The full loop. Project 00 shows you what an environment is; this one takes a single environment all the way to a trained model.
Qwen3-VL-2B learns to read rendered math into LaTeX with GRPO, scored by a reward served from a live OpenEnv Space. Correctness is checkable — render the prediction, compare — so the reward is honest and there is very little to game. Runs on a GPU you spin up in one command; no cluster, no local GPU.
02 · Watercolour train against an aesthetic reward
RL over taste. Project 01 trains against a verifiable reward. This one asks what happens when there is no correct answer at all.
Qwen3.5-35B-A3B learns to paint watercolours by writing p5.brush sketches. Headless Chromium renders each sketch, and the render is scored by an aesthetic preference model plus a VLM judge comparing it against a hand-rated reference pool, so the reward is somebody's taste. Three reward mixes trained and compared, every artifact published, and the full story in the blog post.
03 · GeoGuesser train a VLM to place itself on Earth
A game as an environment. Projects 01 and 02 score one shot. This one is multi-turn: the agent looks around, walks down the road, zooms on a sign, pins a candidate, and commits.
Qwen3.5-4B learns visual geolocation with GRPO against a
playable OpenEnv Space serving real
Mapillary panoramas, scored on kilometres of error. It ends up ahead of gpt-5.4-mini and
claude-haiku-4.5 on a 200-task held-out split and behind only claude-sonnet-5, for about $100.
Ten hours on four A100s, or one GPU if you are in no hurry. Three training runs, the reward
redesigned once, and every measurement bug written down in
the article.
More coming. Each new project is another end-to-end recipe: an environment, a training run, and the artifacts on the Hub. Proposals and contributions welcome →
Articles & talks
Long-form writing and conference talks. Sources live in content/; each one ships to the
Hub as a Space.
|
How to turn a game into an RL environment
The technical intuition, worked end to end: curating the data, designing the environment, shipping it
with OpenEnv, and training a 4B against it with TRL until it outscored 📂 |
The Ultimate Guide to RL Environments
Building and scaling RL environments in the LLM era — how frameworks are built, how rewards are wired, and how they scale to thousands of concurrent sessions. |
RL Environments 101
From "what is an env?" to training your own. RL fundamentals → environment anatomy → OpenEnv → training with TRL. The original 30-minute talk. |
Scaling RL for LLMs
What an environment actually is, how reward hacking happens, and how to build and train against your own. The 20-minute cut, for AMD AI Dev Day. |
|
Multi-Harness Training
OpenEnv × Harbor — why an environment's failure model decides whether it can be trained against: in-process agent loops vs. an HTTP boundary, and what it takes to capture trainable tokens. |
Quickstart
Run an environment. Wordle is pure Python with no external backend — the fastest full rollout:
git clone https://github.com/adithya-s-k/HuggingEnvs
cd HuggingEnvs
cp .env.example .env # HF_TOKEN, plus E2B_API_KEY for sandbox-backed envs
cd 00-environments-101/envs/wordle/verifiers
uv sync && uv run python rollout.py
Train a model against one. No GPU, no cluster, no setup — one command spins up a GPU with the notebooks loaded and prints a JupyterLab URL:
curl -sSL https://raw.githubusercontent.com/adithya-s-k/HuggingEnvs/main/tools/jupyter_launch.py | python3 -
Windows (PowerShell): irm https://raw.githubusercontent.com/adithya-s-k/HuggingEnvs/main/tools/jupyter_launch.py | python -. Set FLAVOR=t4-small for a cheaper GPU. Track jobs at huggingface.co/settings/jobs.
Build your own environment
Five SKILL.md-spec agent skills turn a plain-English description into a runnable RL environment across four frameworks. They work in any project — with Claude Code, Cursor, Codex, OpenCode, Gemini CLI and others.
npx skills add adithya-s-k/HuggingEnvs
| Skill | What it builds |
|---|---|
rl-env-from-description | Orchestrator — interviews you, then ports across all four frameworks |
generate-openenv-env | OpenEnv (Hugging Face / Meta) — HTTP + MCP |
generate-ors-env | OpenReward (ORS) — per-tool-call rewards |
generate-verifiers-env | Verifiers (Prime Intellect) — in-process + rubrics |
generate-nemo-gym-env | NeMo Gym (NVIDIA) — HTTP + post-episode /verify |
"make me an env where the agent plays connect-four" — that's the whole interface.
Repository layout
HuggingEnvs/
├── 00-environments-101/ 3 environments × 6 frameworks
├── 01-latex-ocr/ train a VLM against a served reward
├── content/
│ ├── articles/ long-form sources (Astro → Docker Space)
│ └── slides/ talk decks (Vite → static Space)
├── tools/ launcher, Space deploy, index generation
├── assets/ brand + content thumbnails
└── .claude/skills/ the five environment-authoring agent skills
Inside a project the folders always mean the same thing: envs/ (implementations, shared logic in
core/), train/ (configs + launch), notebooks/, results/.
Contributing
We're actively looking for new end-to-end recipes — a task, an environment, a training run, and honest results. Domains we don't cover yet are especially welcome: web browsing, SQL, games, robotics sims, tool-use over real APIs, long-horizon software engineering.
Half-finished counts. A recipe with real numbers and a gap beats a polished one nobody ran — open an issue and we'll help you land it. New framework ports, reproductions that disagree with ours, and corrections to the guide are all just as welcome.
See CONTRIBUTING.md. The fastest path to a new environment is the
rl-env-from-description skill.
Citation
@misc{huggingenvs,
author = {Kolavi, Adithya S},
title = {HuggingEnvs: Open Source RL Environments for LLM Agents},
year = {2026},
url = {https://github.com/adithya-s-k/HuggingEnvs}
}
License
Similar repos
Contributors
81 commits
15 commits
Languages
Python
22.2%
JavaScript
21.9%
HTML
20.2%
TypeScript
12.7%
Astro
8.9%
TeX
6.7%
CSS
3.1%
MDX
2.5%