Forge Agentic Workflow Evaluation Corpus

Forge evaluates multi-step agent workflows across models, quantizations, inference backends, function-calling modes, scenarios, guardrail ablations, and reasoning-replay policies. This dataset publishes the released run-level outcome records behind Forge's reports and dashboard. One row represents one attempted evaluation run.

This is an outcome corpus, not a collection of complete agent traces. It does not contain full prompts, conversations, tool-result transcripts, or hidden model reasoning and cannot reconstruct an end-to-end trajectory.

Configurations and counts

• latest is the default and contains selected rows at the corpus-wide maximum evaluation generation.
• snapshot retains the selected arm for each comparable configuration, including older carried evidence when no newer matching arm exists.
• history contains every released row, including superseded generations, in pinned source-file and source-line order.

Dataset repository: https://huggingface.co/datasets/antoinezambelli/forge-evals

Quickstart

from datasets import load_dataset

latest = load_dataset("antoinezambelli/forge-evals", "latest", split="train")
snapshot = load_dataset("antoinezambelli/forge-evals", "snapshot", split="train")
history = load_dataset("antoinezambelli/forge-evals", "history", split="train")

Metric contract

The canonical headline metric is Score:

• score = correct_count / attempted_count
• validated_accuracy = correct_count / validated_count
• completion_rate = completed_count / attempted_count

attempted_count means rows present in the selected cohort, not a theoretical schedule. correct is true, false, or null when no usable correctness judgment exists. A null judgment remains in the Score denominator and is excluded from the validated-accuracy denominator. completed records whether the workflow returned normally and is independent of correctness. Exact integer components are included in the publication plan so every displayed rate is reproducible without reverse-engineering rounded percentages.

Schema overview

All 52 columns use one explicit v2 schema:

• Identity and condition: model, backend, mode, ablation, tool choice, reasoning replay/level, scenario, run index, and evaluation generation.
• Outcome: correct, completed, validation_error, execution error type, and execution error message.
• Efficiency: iterations, ideal iterations, wasted calls, elapsed seconds, context budget, and stream retries.
• Guardrail and reasoning telemetry: nudges, tool errors, compaction events, captured reasoning counts, and on-wire reasoning counts.
• Hosted accounting: input, output, cache-creation, cache-read tokens, and recorded cost when the source backend supplied them.
• Provenance and selection: release, source file and line hashes, generation metadata, canonical configuration/arm identifiers, selection status, and view membership.

The builder never modifies source JSONLs. Each accepted source revision is pinned by its exact hash, so intentional provenance corrections require an explicit pin update. Legacy accuracy/completeness spelling is normalized to published correct/completed; the legacy aliases do not appear in the Parquet schema. Sparse source fields become null.

Generations, replay, and carried evidence

An evaluation generation is a comparability epoch, not generated model text and not necessarily a Forge release. A generation changes when collection semantics materially change. Several releases may share one generation.

The raw reasoning_replay field preserves the source value. Rows predating the knob have no raw value and resolve to effective full; this legacy-inferred arm remains distinct from an explicitly recorded full arm. Missing raw reasoning_level resolves to effective default. Carried evidence is an older selected cohort retained only because no newer matching cohort exists.

Methodology and statistical use

Forge scenarios exercise tool selection, argument fidelity, multi-step sequencing, error recovery, stateful interactions, and context-pressure paths. Repeated runs are stored individually. Cohort comparisons should control for model, quantization, backend, mode, scenario set, replay policy, reasoning level, generation, and collection environment. For paired arms, Forge uses paired McNemar tests on matching (scenario, run) observations and Wilson intervals for Score. The run rows contain the paired observations and exact outcome components needed to recompute those analyses independently. This bundle does not include a packaged Parquet-to-report or dashboard command.

Intended uses

• Recompute aggregate metrics and perform independent analyses from run-level outcomes.
• Compare controlled model/backend/mode or ablation cohorts.
• Study completion, validation, efficiency, replay, and guardrail behavior.
• Build new statistical views while retaining source-level provenance.

This dataset should not be used as a general model-quality leaderboard, as training text, or as evidence that one backend/model is universally superior. It also does not measure subjective answer quality beyond each scenario's deterministic validator.

Limitations

Many cells are intentionally absent or inapplicable, and a missing scenario is not scored as an attempted failure. Historical evidence spans different rigs, backends, quantizations, serving versions, context budgets, and model-specific reasoning behavior. Timing, token, and cost fields are operational measurements and are only comparable under compatible collection conditions. Repeated runs within one scenario may be more correlated than independent scenario-level effects.

Provenance and integrity

provenance/sources.json pins every released source, generation, dialect, row count, and SHA-256. provenance/schema.json contains the complete source-dialect and normalized-schema contract. manifest.json records every configuration, logical digest, shard row count, file hash, builder-source hash, Git revision, Python version, and Parquet writer version. Verify a downloaded bundle with:

python -m tests.eval.dataset_builder verify --source-root . --bundle <bundle>

Project and methodology: https://github.com/antoinezambelli/forge/blob/main/docs/EVAL_GUIDE.md

Paper citation: Zambelli, A. Forge: Closing the Agentic Reliability Gap Between Self-Hosted and Frontier Language Models. https://doi.org/10.1145/3786335.3813193

License: mit