README
FIRE-Bench (verified)
A benchmark of 35 hand-curated research tasks from the FIRE-Bench
project. Unlike the auto-generated companion dataset
silence-suzuki/FIRE-Bench-unverified,
these have been written and reviewed manually -- prompts, ground-truth
plans, and conclusions are all human-validated.
Schema
| field | description |
|---|---|
task_id | unique identifier (e.g. activation_control) |
research_question | the question the agent must answer |
instruction | full prompt the agent sees (research question + resources) |
instruction_gt | ground-truth procedural plan (used for evaluation, not shown to the agent) |
conclusion | ground-truth answer; what the agent's final write-up is compared against |
dataset_source | upstream URL or short note for the data, when the curators left one |
has_local_data | true when raw data files are bundled at tasks/<task_id>/data/ in this repo |
Usage
from datasets import load_dataset
ds = load_dataset("silence-suzuki/FIRE-Bench-verified", split="train")
for task in ds:
output = my_agent.run(task["instruction"])
# evaluate output against task["conclusion"]
The raw per-task files (instruction.txt, instruction_gt.txt,
conclusion.txt) are also available under tasks/<task_id>/ for
filesystem-walking workflows. For tasks where the curators bundled local
data, tasks/<task_id>/data/ holds the dataset files (JSONL, JSON,
images, etc.) and tasks/<task_id>/dataset.txt documents the source.
To pull just the assets for one task:
from huggingface_hub import snapshot_download
snapshot_download(
"silence-suzuki/FIRE-Bench-verified",
repo_type="dataset",
allow_patterns=["tasks/lost_in_the_middle/*", "tasks/lost_in_the_middle/**"],
)
For the runner / scoring code see maitrix-org/FIRE-Bench.
Contributors
5 commits
Similar repos
Linked in READMEs
README
FIRE-Bench (verified)
A benchmark of 35 hand-curated research tasks from the FIRE-Bench
project. Unlike the auto-generated companion dataset
silence-suzuki/FIRE-Bench-unverified,
these have been written and reviewed manually -- prompts, ground-truth
plans, and conclusions are all human-validated.
Schema
| field | description |
|---|---|
task_id | unique identifier (e.g. activation_control) |
research_question | the question the agent must answer |
instruction | full prompt the agent sees (research question + resources) |
instruction_gt | ground-truth procedural plan (used for evaluation, not shown to the agent) |
conclusion | ground-truth answer; what the agent's final write-up is compared against |
dataset_source | upstream URL or short note for the data, when the curators left one |
has_local_data | true when raw data files are bundled at tasks/<task_id>/data/ in this repo |
Usage
from datasets import load_dataset
ds = load_dataset("silence-suzuki/FIRE-Bench-verified", split="train")
for task in ds:
output = my_agent.run(task["instruction"])
# evaluate output against task["conclusion"]
The raw per-task files (instruction.txt, instruction_gt.txt,
conclusion.txt) are also available under tasks/<task_id>/ for
filesystem-walking workflows. For tasks where the curators bundled local
data, tasks/<task_id>/data/ holds the dataset files (JSONL, JSON,
images, etc.) and tasks/<task_id>/dataset.txt documents the source.
To pull just the assets for one task:
from huggingface_hub import snapshot_download
snapshot_download(
"silence-suzuki/FIRE-Bench-verified",
repo_type="dataset",
allow_patterns=["tasks/lost_in_the_middle/*", "tasks/lost_in_the_middle/**"],
)
For the runner / scoring code see maitrix-org/FIRE-Bench.
Similar repos
Linked in READMEs
Contributors
5 commits