OpenMLE Tasks

📄 Paper  •  🌐 Project  •  💻 Code  •  🤗 Models  •  📚 SFT Traces

OpenMLE Tasks provides machine-learning Task environments. The public SFT trajectories are released separately in OpenMLE-SFT-Traces. These resources accompany the paper Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineering and the OpenRSI code release.

Layout

recipes/<task_id>/<task_key>/       # scripts for rebuilding a recipe Task
tasks/<task_id>/package.json        # archive parts, sizes, and SHA-256 values
tasks/<task_id>/task.tar.zst        # complete built Task for a single-part archive
tasks/<task_id>/task.tar.zst.part-* # ordered parts of a split archive, when present
package_manifest.jsonl              # all Task archive sizes, SHA-256 values, and source statistics
task_index.jsonl                    # Task index, upstream links, and package locations
build_task.py                       # one command for both release forms
requirements.txt                    # recipe-reconstruction dependencies

Download the launcher files

Python 3.10 or later and uv are required. Install uv if needed:

python -m pip install uv

You do not need to clone the complete dataset repository to materialize one Task. Download the launcher and index files, then enter the resulting directory:

uvx --from 'huggingface_hub>=0.34' hf download FrontisAI/OpenMLE-Tasks \
  build_task.py task_index.jsonl requirements.txt \
  --repo-type dataset \
  --local-dir OpenMLE-Tasks
cd OpenMLE-Tasks

If you clone the complete repository with Git instead, install both Git Xet and Git LFS before cloning. Otherwise, tracked dataset files may remain small pointer files rather than their actual contents.

git xet install
git lfs install
git clone https://huggingface.co/datasets/FrontisAI/OpenMLE-Tasks

Rebuild a Task after accepting upstream terms

1. Find the Task in task_index.jsonl and open every listed source_urls page.
2. On the upstream page, sign in and click to accept the applicable licence or competition terms. The script cannot accept terms for you.
3. Log in to the platform CLI when required (kaggle or aicrowd).
4. Run:

uv run --with-requirements requirements.txt python build_task.py \
  --task-key '<task_key>' \
  --output-dir /absolute/path/to/generated-task \
  --accept-upstream-terms

--accept-upstream-terms confirms that you accepted the upstream terms; it cannot accept them on your behalf. If automatic download is unavailable, first download the data yourself and add --raw-dir /absolute/path/to/upstream/raw.

Materialize a ready-to-use Task

Run the same entry point without --accept-upstream-terms. Only the archive download and extraction dependencies are needed:

uv run \
  --with 'huggingface_hub>=0.34' \
  --with 'zstandard>=0.23' \
  python build_task.py \
  --task-key '<task_key>' \
  --output-dir /absolute/path/to/generated-task

For a built_task entry, the command reads tasks/<task_id>/package.json, downloads that Task's task.tar.zst or ordered binary parts, verifies every part's SHA-256, and safely extracts the complete Task. The generated directory contains data/public, data/private, the evaluator in utils/metric.py, and its metadata/runtime files.

The built data is stored in archives because some Tasks contain more than 10,000 files in a single directory, which cannot be represented directly in a Hugging Face repository folder. Archives contain ordinary files only; symbolic links and hard links are rejected during release preparation.

Small Tasks have one task.tar.zst in their own Task directory. Archives over 15 GiB are split into ordered task.tar.zst.part-xxxxx-of-yyyyy files in the same directory; they still represent one continuous archive and are joined by build_task.py while streaming. No top-level package directory is published.

checksums.sha256 covers repository metadata and runtime files. Archive-part hashes and compressed sizes are recorded in each Task's package.json and in the root package_manifest.jsonl.

Licenses and upstream terms

This collection contains material under different licences, so the Hugging Face metadata uses license: other.

• Original OpenMLE curation, documentation, and release tooling are licensed under CC BY-NC 4.0 as stated in LICENSE.
• Third-party Task material and upstream data are not relicensed by OpenMLE.
• Upstream links and available licence information are recorded in task_index.jsonl and each Task's RELEASE_METADATA.json.
• Upstream terms control the use and redistribution of third-party material.

Use a Task in OpenMLE-Evo

Both release forms produce a complete Task package. After downloading a built package or rebuilding a recipe from upstream data, prepare the input read by Evo. Keep the complete package, including info/, data/ and utils/.

Make the package available to the sandbox controller and workers at the same container-visible path. The sandbox/router does not transfer local task files. For a package mounted at /mnt/tasks/titanic, use:

The submit setting is the parent directory: Evo appends the task directory name. Neither setting points directly to data/public or data/private. The presence of test_answer.csv does not determine the evaluation phase. Using the same package for both phases reuses the same answers; it does not create an independent held-out test set.

Clone the code and generate the evaluation input:

git clone https://github.com/FrontisAI/OpenRSI.git
cd OpenRSI/OpenMLE-Gym
uv sync --no-editable --extra input
uv run --no-editable --extra input openmle-task prepare-input \
  --task-dir /absolute/host/path/tasks/titanic \
  --sandbox-task-dir /mnt/tasks/titanic \
  --output ../OpenMLE-Evo/artifacts/gym-example/eval.parquet

Replace the example task name and paths with your package location and sandbox mount. The command reads the package metadata and descriptions and uses the bundled prompt template to write Evo's existing prompt/metadata Parquet format. Set OPENMLE_EVAL_DATA to the generated Parquet and OPENMLE_SUBMIT_DATA_DIR_ROOT to the sandbox-visible parent directory.

Follow the Gym task usage guide for sandbox mounts, model configuration, Evo installation and launch commands.

Use Tasks for SFT rollout and RL

The same prepared prompt/metadata Parquet can be consumed by SFT trajectory creation and RL. Set OPENMLE_PARALLEL_DATA for parallel SFT rollout, OPENMLE_EVOLUTIONARY_DATA for evolutionary SFT rollout, and PROMPT_DATA / EVAL_PROMPT_DATA for RL training/evaluation selections. Evolutionary SFT uses OPENMLE_TASK_DATA_ROOT for the final-scoring package parent; this differs from Evo's OPENMLE_SUBMIT_DATA_DIR_ROOT.

See Prepare task inputs for multi-task input preparation and links to each consumer's setup instructions. SFT training consumes generated trajectories containing messages, not the Task input Parquet. Prepare independent task selections where required; input preparation does not generate train/test splits or configure training resources.