CL-PRM Artifacts

Public artifact repository repository for the CL-PRM Course Project. It stores generated experiment data and analysis outputs that are excluded from the GitHub source repository.

Repository Roles

The GitHub repository is the source of truth for code, experiment configurations, frozen evaluation protocols, split manifests and hashes, documentation, and tests. This Hugging Face repository is the source of truth for generated verifier predictions, inference metadata, runtime measurements, router evaluations, tables, statistics, and figures.

Find the Report Results

The fixed 0.5 PathFinder condition is the frozen main experiment. MathPRM and Skywork are exploratory verifier pairs on the same grouped 600/200/400 split. Threshold sensitivity was studied after the main test results were seen; its validation-selected thresholds and test sweeps are descriptive and do not replace the frozen main result.

Current Artifact Layout

cl-prm-artifacts/
├── feasibility/
│   ├── ovm/
│   │   ├── ovm-eval.json
│   │   ├── ovm-eval.md
│   │   └── runs/                       # Three generated seed runs and metadata
│   └── prm_router/
│       ├── pilot_100/                 # Frozen 100-example exploratory pilot
│       ├── expanded_200/             # Development-only 200-example study
│       └── robustness/               # Development robustness analyses
├── formal/
│   ├── train/                         # 600 examples
│   │   ├── reasoneval.jsonl
│   │   ├── reasoneval.metadata.json
│   │   ├── pathfinder.jsonl
│   │   ├── pathfinder.metadata.json
│   │   ├── genprm.jsonl
│   │   └── genprm.metadata.json
│   ├── validation/                    # 200 examples; same verifier filenames
│   └── test/                          # 400 examples; ReasonEval and PathFinder
│       ├── reasoneval.jsonl
│       ├── reasoneval.metadata.json
│       ├── pathfinder.jsonl
│       └── pathfinder.metadata.json
├── evaluation/
│   └── router/
│       ├── reasoneval_threshold.json
│       ├── pathfinder_development_selection.json
│       ├── formal_test_results.json
│       └── final_analysis/
│           ├── final_statistics.json
│           ├── analysis_summary.md
│           ├── quantitative_results.tex
│           ├── main_results.csv
│           ├── paired_comparisons.csv
│           ├── routing_curves.csv
│           ├── call_outcomes_20pct.csv
│           ├── overlap_analysis.csv
│           ├── router_discrimination.csv
│           ├── calibration_summary.csv
│           ├── calibration_bins.csv
│           ├── pre_specified_diagnostics.csv
│           └── figures/              # PDF figures for budget, runtime, calls, calibration
└── extensions/
    ├── full/
    │   ├── train/                   # MathPRM and Skywork predictions and metadata
    │   ├── validation/              # Same verifiers on validation
    │   ├── test/                    # Same verifiers on test
    │   └── evaluation/              # Development selections and test evaluations
    └── threshold_sensitivity/
        ├── comparison_summary.csv
        ├── pathfinder/
        ├── math_prm/
        ├── skywork_prm/
        └── figures/                # Validation/test sweep data and PDF/PNG figures

The formal verifier naming convention is formal/<split>/<verifier>.jsonl with a corresponding formal/<split>/<verifier>.metadata.json. These are files directly under each split directory, not nested verifier directories. The directory above describes the current repository layout; download cache metadata under .cache/huggingface/ is local to a downloaded copy and is not part of the shared artifact structure.

MathPRM and Skywork predictions follow extensions/full/<split>/<verifier>.jsonl with matching metadata. The threshold study's comparison_summary.csv compares fixed 0.5 with thresholds selected by validation balanced accuracy; reproduction_gate.json verifies the original fixed-threshold results before the post-hoc comparisons are interpreted. The pinned split configuration and reconstruction instructions are in the GitHub repository.

Frozen Primary Study

The completed primary experiment uses ReasonEval as the base verifier and PathFinder-PRM as the optional second-stage verifier. The original-question-grouped train/validation/test splits contain 600/200/400 examples. Thresholds and utility weights were selected on development data before the one-time held-out evaluation.

At the frozen 20% PathFinder call budget, expected-gain and cost-aware routing each achieved 81.75% held-out accuracy, compared with 71.75% for ReasonEval-only and 79.00% for PathFinder-only. The full result and grouped-bootstrap intervals are recorded in evaluation/router/formal_test_results.json.

The generated descriptive analysis is in evaluation/router/final_analysis/. Its final_statistics.json records a PASS reproduction gate against the frozen formal evaluator. Router discrimination, calibration, and overlap diagnostics are descriptive; they do not retune the frozen model or primary comparison. The overlap comparison is post-hoc descriptive analysis.

GenPRM-7B train and validation outputs are stored alongside the formal split artifacts, but use an analysis_without_code_execution inference mode. There is no GenPRM held-out test artifact here. These outputs are exploratory extension material and must not be presented as a formal official-path GenPRM test result.

Reproducibility and Storage Policy

Each formal inference output has metadata for the model and revision, input/output SHA256 hashes, command, environment, GPU, timing protocol, and completion status. Verify these fields before downstream analysis. The tracked split manifest and formal configuration are in GitHub, while the generated predictions and analyses remain here.

Do not upload model checkpoints that can be fetched from pinned upstream revisions, authentication tokens, unrelated personal data, or local cache files. Run the frozen formal analysis using the GitHub analysis instructions. The course paper is maintained separately in the team's writing workspace.