amstrongzyf/Gome-GPT5-Traces

Dataset

Dataset: GPT-5 Kaggle Agent Traces (Gome)

1

5 commits

1 linked in READMEs

updated Mar 13, 2026

See the code

README

Dataset: GPT-5 Kaggle Agent Traces (Gome)

This folder contains the raw parallel-trace execution logs from the Gome (GPT-5, 12 h, 1*V100) experiments reported in:

Reasoning as Gradient: Scaling MLE Agents Beyond Tree Search [Paper]

The three files here correspond to three of those traces running across 40 Kaggle competitions. Each trace records the full hypothesis β†’ code β†’ execution β†’ feedback loop.

Note: These are raw per-trace logs and do not include the final multi-seed selection step (Β§3.6 of the paper), where top-k candidates are re-run with multiple seeds before submission. The leaderboard scores reported in the paper reflect that additional step.

JSON Structure

Each file shares the same nested structure:

{
  "<competition-name>": {
    "scenario": { ... },      ← competition description & environment
    "loop_0":   { ... },      ← first evolution loop
    "loop_1":   { ... },      ← second evolution loop
    ...
  },
  ...
}

scenario β€” Competition description

Metadata describing the competition environment seen by the agent before it starts.

FieldTypeDescription
backgroundstrStructured competition description fed to the agent (task type, data description, metric)
submission_specificationsstrRequired output format for submission
evaluationstrDetailed metric definition
metric_namestrPrimary evaluation metric name (e.g., "Multi-class Logarithmic Loss")
metric_directionboolfalse = lower is better, true = higher is better
raw_descriptionstrOriginal Kaggle competition page (Markdown); not used when use_raw_description=false
runtime_environmentstrPython version, GPU info, CUDA version of the execution sandbox
renderedstrPre-formatted prompt combining background + specs + evaluation + environment
rewrite_scenstrOne-sentence competition summary used internally

loop_N β€” One evolution iteration

Each loop corresponds to one hypothesis β†’ code β†’ execution β†’ feedback cycle. Fields marked core are present in every loop; others are present in ~82–93% of loops (absent when a loop timed out or was rejected before execution).

FieldTypePresentDescription
hypotheses_rewritedictcore2–3 candidate hypotheses proposed in this loop, each with component tag (FeatureEng / Model / Workflow), rationale, and multi-dimensional scores (alignment, impact, novelty, feasibility, risk/reward)
final_hypothesisdictcoreThe hypothesis selected for implementation; includes component, problem description, and problem label (SCENARIO_PROBLEM / EXECUTION_ERROR / etc.)
taskdictcoreDetailed step-by-step implementation specification generated from the final hypothesis; includes description and package_info
base_codestrcoreStarting code inherited from the previous best iteration (empty string on loop_0)
sota_hypothesisdict~90–93%Best-performing hypothesis seen so far up to this loop
codestr~82–90%Full Python main.py generated and executed by the agent
running_timefloat~82–90%Execution wall time in seconds
valid_scoredict~82–90%Validation score(s) reported by the code: {metric_name: {model_name: score}}
test_reportdict~82–90%Kaggle leaderboard result: public score, medal thresholds, above/below-median flag
feedbackdict~82–90%Evaluator's structured feedback: decision (accept/reject), reason, observations, hypothesis_evaluation, new_hypothesis, code_change_summary

agent
code
machine_learning_engineering

Contributors

amstrongzyf

5 commits

amstrongzyf/Gome-GPT5-Traces

Dataset

Dataset: GPT-5 Kaggle Agent Traces (Gome)

1

5 commits

1 linked in READMEs

updated Mar 13, 2026

See the code

README

Dataset: GPT-5 Kaggle Agent Traces (Gome)

This folder contains the raw parallel-trace execution logs from the Gome (GPT-5, 12 h, 1*V100) experiments reported in:

Reasoning as Gradient: Scaling MLE Agents Beyond Tree Search [Paper]

The three files here correspond to three of those traces running across 40 Kaggle competitions. Each trace records the full hypothesis β†’ code β†’ execution β†’ feedback loop.

Note: These are raw per-trace logs and do not include the final multi-seed selection step (Β§3.6 of the paper), where top-k candidates are re-run with multiple seeds before submission. The leaderboard scores reported in the paper reflect that additional step.

JSON Structure

Each file shares the same nested structure:

{
  "<competition-name>": {
    "scenario": { ... },      ← competition description & environment
    "loop_0":   { ... },      ← first evolution loop
    "loop_1":   { ... },      ← second evolution loop
    ...
  },
  ...
}

scenario β€” Competition description

Metadata describing the competition environment seen by the agent before it starts.

FieldTypeDescription
backgroundstrStructured competition description fed to the agent (task type, data description, metric)
submission_specificationsstrRequired output format for submission
evaluationstrDetailed metric definition
metric_namestrPrimary evaluation metric name (e.g., "Multi-class Logarithmic Loss")
metric_directionboolfalse = lower is better, true = higher is better
raw_descriptionstrOriginal Kaggle competition page (Markdown); not used when use_raw_description=false
runtime_environmentstrPython version, GPU info, CUDA version of the execution sandbox
renderedstrPre-formatted prompt combining background + specs + evaluation + environment
rewrite_scenstrOne-sentence competition summary used internally

loop_N β€” One evolution iteration

Each loop corresponds to one hypothesis β†’ code β†’ execution β†’ feedback cycle. Fields marked core are present in every loop; others are present in ~82–93% of loops (absent when a loop timed out or was rejected before execution).

FieldTypePresentDescription
hypotheses_rewritedictcore2–3 candidate hypotheses proposed in this loop, each with component tag (FeatureEng / Model / Workflow), rationale, and multi-dimensional scores (alignment, impact, novelty, feasibility, risk/reward)
final_hypothesisdictcoreThe hypothesis selected for implementation; includes component, problem description, and problem label (SCENARIO_PROBLEM / EXECUTION_ERROR / etc.)
taskdictcoreDetailed step-by-step implementation specification generated from the final hypothesis; includes description and package_info
base_codestrcoreStarting code inherited from the previous best iteration (empty string on loop_0)
sota_hypothesisdict~90–93%Best-performing hypothesis seen so far up to this loop
codestr~82–90%Full Python main.py generated and executed by the agent
running_timefloat~82–90%Execution wall time in seconds
valid_scoredict~82–90%Validation score(s) reported by the code: {metric_name: {model_name: score}}
test_reportdict~82–90%Kaggle leaderboard result: public score, medal thresholds, above/below-median flag
feedbackdict~82–90%Evaluator's structured feedback: decision (accept/reject), reason, observations, hypothesis_evaluation, new_hypothesis, code_change_summary

agent
code
machine_learning_engineering

Contributors

amstrongzyf

5 commits