README
AweAgent-Meta-NL2Repo
This dataset provides the metadata used by AweAgent to run the NL2RepoBench evaluation.
If you are looking for the underlying benchmark itself (task design, repositories, test suites), please refer to the original project: multimodal-art-projection/NL2RepoBench.
Purpose
The AweAgent repository evaluates end-to-end repo-level code generation: given a natural-language project specification, the agent must produce a working Python package that passes the project's tests. To run that evaluation reproducibly, the harness needs a compact, machine-readable manifest of every instance — that is exactly what this dataset provides.
Concretely, each row tells AweAgent:
- which prebuilt evaluation Docker image to launch,
- the NL prompt that defines the target repository,
- the verification command and the files that must be produced,
- the number of test cases the generated repo will be scored against.
Files
nl2repo_aweagent.jsonl— one JSON object per NL2RepoBench instance (104 instances).
Schema
| Field | Type | Description |
|---|---|---|
instance_id | str | Unique identifier for the instance (typically the target package name). |
package_name | str | The Python package the agent is expected to generate. |
evaluation_image | str | Docker image (hosted under ghcr.io/multimodal-art-projection/nl2repobench) used to evaluate the generated repository. |
start_instruction | str | The natural-language task description handed to the agent as the starting prompt. |
verify_files | list[str] | Files that must be produced by the agent and are checked during verification. |
verify_cmd | str | The command executed inside the evaluation image to verify the generated repository. |
test_cases_num | int | Number of test cases used to score the instance. |
Usage
from datasets import load_dataset
ds = load_dataset("AweAI-Team/AweAgent-Meta-NL2Repo", split="train")
print(ds[0])
This manifest is consumed by the evaluation pipeline in AweAgent; see that repository for the full runner, scoring logic, and reproduction instructions.
Acknowledgements
This dataset is built on top of, and would not exist without, the excellent NL2RepoBench project by the Multimodal Art Projection team. All benchmark instances, evaluation images, and test cases originate from their work; this dataset only repackages the per-instance metadata in the form AweAgent's evaluation harness expects. Huge thanks to the NL2RepoBench authors for releasing such a high-quality repository-level code-generation benchmark.
License
Released under CC BY 4.0. When using this dataset, please also cite and credit the upstream NL2RepoBench project.
Contributors
3 commits
Similar repos
README
AweAgent-Meta-NL2Repo
This dataset provides the metadata used by AweAgent to run the NL2RepoBench evaluation.
If you are looking for the underlying benchmark itself (task design, repositories, test suites), please refer to the original project: multimodal-art-projection/NL2RepoBench.
Purpose
The AweAgent repository evaluates end-to-end repo-level code generation: given a natural-language project specification, the agent must produce a working Python package that passes the project's tests. To run that evaluation reproducibly, the harness needs a compact, machine-readable manifest of every instance — that is exactly what this dataset provides.
Concretely, each row tells AweAgent:
- which prebuilt evaluation Docker image to launch,
- the NL prompt that defines the target repository,
- the verification command and the files that must be produced,
- the number of test cases the generated repo will be scored against.
Files
nl2repo_aweagent.jsonl— one JSON object per NL2RepoBench instance (104 instances).
Schema
| Field | Type | Description |
|---|---|---|
instance_id | str | Unique identifier for the instance (typically the target package name). |
package_name | str | The Python package the agent is expected to generate. |
evaluation_image | str | Docker image (hosted under ghcr.io/multimodal-art-projection/nl2repobench) used to evaluate the generated repository. |
start_instruction | str | The natural-language task description handed to the agent as the starting prompt. |
verify_files | list[str] | Files that must be produced by the agent and are checked during verification. |
verify_cmd | str | The command executed inside the evaluation image to verify the generated repository. |
test_cases_num | int | Number of test cases used to score the instance. |
Usage
from datasets import load_dataset
ds = load_dataset("AweAI-Team/AweAgent-Meta-NL2Repo", split="train")
print(ds[0])
This manifest is consumed by the evaluation pipeline in AweAgent; see that repository for the full runner, scoring logic, and reproduction instructions.
Acknowledgements
This dataset is built on top of, and would not exist without, the excellent NL2RepoBench project by the Multimodal Art Projection team. All benchmark instances, evaluation images, and test cases originate from their work; this dataset only repackages the per-instance metadata in the form AweAgent's evaluation harness expects. Huge thanks to the NL2RepoBench authors for releasing such a high-quality repository-level code-generation benchmark.
License
Released under CC BY 4.0. When using this dataset, please also cite and credit the upstream NL2RepoBench project.
Similar repos
Linked in READMEs
Contributors
3 commits