Bootstrapping Reasoning in Compact Language Models

A Multi-Stage Reinforcement Learning Pipeline with Targeted Failure Repair

Joseph Li · Max Rodriguez · Victor Chen — Stanford University Stanford CS224N: Natural Language Processing with Deep Learning

Paper in progress. Full manuscript available upon request.

---

Overview

We train a 0.5B-parameter model (Qwen2.5-0.5B-Instruct) to solve competition-level math problems using reinforcement learning from verifiable rewards alone — no teacher model, no synthetic reasoning traces. Our three-stage pipeline achieves 20.8% overall accuracy on the Hendrycks MATH algebra benchmark, surpassing a knowledge-distilled baseline (20.3%) that requires a 32B teacher model.

On Level 1 algebra problems, the final model reaches 56.3% accuracy — a +21.5 point improvement over SFT alone.

Key Results

ThinkPRM evaluated on 100 examples due to compute cost.

See RESULTS.md for full tables, per-level breakdowns, and CaSE reasoning quality scores.

---

Pipeline

Qwen2.5-0.5B-Instruct (base)
         │
         ▼
┌─────────────────────┐
│  Stage 1: SFT       │  Full-weight fine-tuning on ground-truth solutions.
└─────────┬───────────┘  Establishes math notation, formatting, \boxed{} convention.
          │  14.6% accuracy
          ▼
┌─────────────────────┐
│  Stage 2: GRPO      │  Self-play RL with verifiable answer-correctness rewards.
└─────────┬───────────┘  Group size G=8, KL coeff 0.005, entropy bonus 0.01.
          │  +2.9% from self-generated exploration
          ▼
┌──────────────────────────────┐
│  Failure Detection           │  Sample 8 completions per problem:
└─────────┬────────────────────┘  mastered (8/8) · borderline (1–7/8) · hard fail (0/8)
          ▼
┌─────────────────────────────────────────────────────┐
│  Stage 3: Targeted DAPO Repair                      │  DAPO on failures + borderline only.
└─────────────────────────────────────────────────────┘  LoRA r=16, 30% replay buffer.
          │  +3.3% by concentrating compute on frontier problems
          ▼
    Final model: 20.8% — surpasses distillation without any teacher

At inference time, pairing the final model with a compact ThinkPRM verifier (Best-of-8 selection) further boosts accuracy to 27.0% (+6.2 points at no additional training cost).

---

Repository Structure

CS224n/
├── pipeline/          # Three-stage RL training: SFT → GRPO → Targeted DAPO Repair  [§4.1]
├── distillation/      # Knowledge-distillation baseline (Gemini 2.5-Flash teacher)    [§4.3]
├── verification/      # ThinkPRM and RM-R1 process reward model training              [§4.2]
├── evaluation/        # Best-of-N selection, CaSE reasoning quality, verifier compare [§5–6]
├── data/              # Hendrycks MATH stratified splits and dataset scripts           [§5]
└── utils/             # Shared Google Cloud Storage utilities

---

Quick Start

# 1. Environment
conda env create -f environment.yml        # macOS (development)
conda env create -f environment.linux.yml  # Linux/GPU (training)
conda activate cs224n

# 2. Cloud compute
pip install modal && modal setup

# 3. Run the full RL pipeline (~4.8 hours on one A100-80GB)
modal run pipeline/trainers/modal_train_targeted.py --run-tag v1

# 4. Evaluate
modal run pipeline/eval/modal_eval_dapo_algebra.py --run-tag v1

# 5. Best-of-N with ThinkPRM verifier
conda run -n cs224n python evaluation/best_of_n/Best_of_N_selection_eval.py \
    --dapo_model_path <checkpoint> \
    --thinkprm_model_path <verifier_checkpoint> \
    --num_candidates 8

---

Design Findings

Several choices that are routine at larger scales become critical at 0.5B parameters:

---

Infrastructure

• Compute: Modal serverless A100-80GB GPUs
• Storage: Google Cloud Storage for datasets and checkpoints (configure via environment variables — see each module's README)
• Vertex AI: Gemini 2.5-Flash for rationale extraction and CaSE reasoning quality evaluation

---

Citation

This work is currently under development. If you find it useful, please check back for a citable version once the paper is finalized.

Joseph Li, Max Rodriguez, Victor Chen.
"Bootstrapping Reasoning in Compact Language Models:
 A Multi-Stage Reinforcement Learning Pipeline with Targeted Failure Repair."
Stanford CS224N, 2026.