Reasoning-vs-InfernceScaling

This repository contains code to test reasoning capabilities using the Quiet Star model and inference scaling laws with Rebase. Experiments were conducted using:

• 1 A100 GPU
• 8 CPUs

The main purpose of this was to implemnt infernce scaling into a reasoning model (quietstar eval/quiet-rebase.py), but ended up producing worse results than the base model.

To install required libraries

pip install -r requirements.txt

Configuration Settings

General Configuration

REBASE Configuration

Note: Random seed 42 was maintained from the original QuietSTaR implementation to ensure comparable results.

Repository Contents

The repository includes:

• base.py: Evaluates the base Mistral-7B model for:
  • Accuracy
  • Inference time
  • FLOPS per inference
• rebase.py: Same metrics using Rebase for inference optimization
  • Note: To run rebase.py, modify the hyperparameters in rebase.yaml first
• quietstar eval/quietstar.py: Same metrics for Quiet Star model
  • Note: To run quietstar you need to have the custom modeling_mistral and config files in the same directory as the evaluation script
• quietstar eval/quiet-rebase.py: Same metrics using rebase inference with quiet-star model
  • Note: To run quietstar you need to have the custom modeling_mistral and config files in the same directory as the evaluation script

Results folder

The results contain results for basline Mistral7b model, quietstar, rebase with 3,6,and 16 width tree

The evaluation was performed on 128 questions from the GSM8K dataset.

For rebase we conducted different experiments with varying width sizes of the Rebase tree: 3, 6, and 16 (16 was the original width size from the paper).

Efficiency Metrics & Rankings

Equations

1. Accuracy per TFLOP = Accuracy (%) / FLOPs (Trillion)
2. Accuracy per Second = Accuracy (%) / Time (seconds)
3. Efficiency Score = (Accuracy per TFLOP × Accuracy per Second) × 100

Results

Example Calculation (REBASE w=3)

1. Accuracy per TFLOP = 10.94% / 2.35T = 4.66
2. Accuracy per Second = 10.94% / 8.47s = 1.29
3. Efficiency Score = (4.66 × 1.29) × 100 = 601.14