ModularRSI 2000 Instances

Dataset summary

This dataset is the independent evolution pool released with ModularRSI. It contains Harbor-formatted tasks for studying whether an agent can improve its harness from execution experience and transfer those improvements to unseen tasks.

Companion code: IQuestLab/ModularRSI

The 120-task training subset is not duplicated. Each domain's numbered manifest.json marks entries 1–120 as train and entries 121–1000 as pool.

Dataset structure

ModularRSI_2000_Instances/
├── tb/
│   ├── manifest.json
│   └── tasks/<task-name>/
└── swe/
    ├── manifest.json
    └── tasks/<task-name>/

Every task follows Harbor's local task layout and includes a task.toml, an instruction, an environment definition, tests, and a reference solution. Each manifest entry records:

• index: stable position from 1 to 1000;
• name: task directory name;
• split: train or pool;
• tree_sha256: digest of the released task tree;
• files and bytes: task-level size metadata.

Usage

Install the separate ModularRSI code repository, then point its single launcher at either domain:

cd /path/to/ModularRSI
export SUPPORT_DATASET_DIR=/path/to/ModularRSI_2000_Instances/tb
bash scripts/self_evolve.sh evolve tools

The launcher uses the 120 manifest entries marked train by default. Set SUPPORT_SPLIT=all to use all 1,000 tasks. Replace tb with swe for the software-engineering domain.

Source and curation

• tb contains generated terminal tasks curated for independent harness evolution.
• swe is a Harbor-formatted subset derived from AweAI-Team/ScaleSWE.
• Release preparation removed local paths, credentials, static private keys, runtime outputs, annotator contact fields, and generated caches. Unsafe candidates were replaced before the numbered manifests were finalized.
• The dataset contains training tasks only. Held-out benchmark trajectories and evaluation results are released separately from this pool.

Limitations

• Tasks require Harbor and Docker-compatible sandbox execution.
• Some tasks depend on public container images and network availability.
• A task being included does not imply uniform difficulty or guaranteed success for every model and environment.
• The pool split is intended for new evolution experiments, not as a held-out benchmark score.

License

The ModularRSI dataset compilation, selection metadata, and original task material are released under CC BY-NC 4.0 for non-commercial use only. See LICENSE.md.

ScaleSWE-derived material retains its upstream CC BY 4.0 terms. Third-party repositories, dependencies, and container images retain their own licenses. The CC BY-NC notice does not replace or restrict those upstream licenses.

Citation

@misc{modularRSI_blog_2026,
  title     = {Exploration of Harness RSI: Methodology, Generalization, and Empirical Foundations},
  url       = {https://recursive-self-improvement.notion.site/blog-1-modularrsi-toward-generalizable-harness-rsi},
  publisher = {Notion},
  author    = {Wu, Siwei and Ren, Jincheng and Li, Yizhi and Li, Haau-Sing and
               Yang, Chengran and Gu, Weicheng and Zhang, Yuxuan and Yang, Jian and
               Batista-Navarro, Riza and Zhang, Chuanyi and Zhou, Ming and Dai, Bryan and
               Lin, Chenghua},
  year      = {2026},
  month     = {Aug}
}