FIRE-Bench

A benchmark of 153 research tasks auto-generated from 58 academic papers via Paper2Bench. Each task hands an agent a research question plus the resources the original paper used (models, datasets, budget, constraints) and asks it to design and run its own experiments.

What's in each task

Every row contains:

Most datasets in task_config.datasets carry a concrete fetch path:

• source: huggingface -- with a verified load_dataset(...) call
• source: external -- with a HEAD-validated url (GitHub / project page)
• source: synthetic -- with generation code in the loader field
• source: unknown -- locator could not place it; usually paper-internal data

Usage

from datasets import load_dataset

ds = load_dataset("silence-suzuki/FIRE-Bench-unverified", split="train")

for task in ds:
    print(task["task_id"], "->", task["research_question"][:80])

    # The simplest path: feed the instruction to your agent.
    output = my_agent.run(task["instruction"])

    # Or pull structured fields:
    for d in task["task_config"]["datasets"] or []:
        if d["source"] == "huggingface":
            print("  load:", d["loader"].splitlines()[-1])
        elif d["source"] == "external":
            print("  fetch:", d["url"])

The raw per-task files (instruction.txt, instruction_gt.txt, task_config.yaml) are also available under tasks/<task_id>/ for users who prefer to walk the filesystem.

Status

Tasks are unverified -- generated end-to-end by an LLM pipeline without human review. Known caveats:

• The arxiv search may have downloaded a different paper than intended for some entries (we mitigate with title-similarity rerank but it's not perfect).
• Research questions, dataset lists, and ground-truth plans are LLM-extracted.
• ~27% of dataset entries are still source: unknown; the rest have a verified loader (HF / synthetic) or HEAD-validated URL (external).

For evaluation, the companion repo at maitrix-org/FIRE-Bench hosts the agent runner and scoring code.

Generation pipeline

Built with Paper2Bench: paper download -> tree parse -> archetype classify -> resource extraction -> HF-loader validation -> HF Hub auto-resolve -> external URL discovery (LLM + HEAD validation) -> render. See the upstream repo for details.