DataDecide evaluation instances

This dataset contains data for individual evaluation instances from the DataDecide project (publication forthcoming). It shows how standard evaluation benchmarks can vary across many dimensions of model design.

The dataset contains evaluations for a range of OLMo-style models trained with:

• 25 different training data configurations
• 9 different sizes with parameter counts 4M, 20M, 60M, 90M, 150M, 300M, 750M, and 1B
• 3 initial random seeds
• Multiple training checkpoints for each model (~10 to ~50 depending on size)
• The 10 different evaluation tasks from OLMES, using cloze formulation:
  • ARC Challenge, ARC Easy, BoolQ, CSQA, HellaSwag, MMLU (57 subtasks), OBQA, PIQA, Social IQa, Winogrande
• 4 different evaluation methods for ranking model answers

In total there are around 150k model checkpoints and 500M individual evaluation instances.

The cloze formulation (as opposed to the "A/B/C/D" multiple choice format) is used to because these models are generally too small to have mastered that format.

The dataset is organized (after untarring) as follows:

models/
├── model_name/ # training mix used, e.g., "dclm-baseline"
│   ├── size/ # e.g., "150M"
│   │   ├── seed/ # e.g., "seed-14"
│   │   │   └── step/ # model checkpoint, e.g., "step-25000"
│   │   │       ├── arc_challenge-metrics.json
│   │   │       ├── arc_challenge-predictions.jsonl
│   │   │       ├── ...

See the sample-evals directory for one example of each task

The -metrics.json file contains the overall metrics for the task while the -predictions.jsonl file contains the predictions for each instance in the following format where the metric suffixes corresponding to different ways of normalizing the model probabilities when ranking the answer choices (see OLMES for details)

• _raw: Raw probability
• _per_token: log-probability per token
• _per_char: log-probability per character
• _uncond: probability of answer divided by unconditional probability of answer (no question given)

Here is an example of a prediction line with annotations:

{
  "doc_id": 0,  # consecutive instance index
  "native_id": "Mercury_7175875",   # task-specific identifier
  "metrics": {  # Overall metrics
    "predicted_index_raw": 3,  # predicted answer indices
    "predicted_index_per_token": 3, 
    "predicted_index_per_char": 3, 
    "predicted_index_uncond": 1, 
    "correct_choice": 2,   # correct answer index
    "acc_raw": 0,   # accuracies for each method
    "acc_per_token": 0, 
    "acc_per_char": 0, 
    "acc_uncond": 0}, 
  "model_output": [  # list of model outputs for each answer choice
    {  # first answer choice
      "sum_logits": -23.55691146850586,  # sum of logprobs of answer tokens
      "num_tokens": 6,  # number of answer tokens
      "num_tokens_all": 201,  # number of tokens in prompt plus answer
      "is_greedy": false,   # whether the answer was the greedy model completion
      "sum_logits_uncond": -34.12132263183594,  # sum of logprobs for unconditional answer tokens
      "logits_per_token": -3.926151911417643,  # normalized logprobs
      "logits_per_char": -0.7138458020759352, 
      "logits_per_byte": 1.029861798615096, 
      "num_chars": 33  # number of characters in answer choice
    }, ...
  ], 
  "label": 2,  # correct answer index
  "task_hash": "da4d61b1b678cfae04369e8a9c4bed3a",  # hash of task configuration
  "model_hash": "596f2b97e34140bf3c9e37fa70e7a5a2"  # hash of model configuration
}

In addition, the dataset contains a summary-metrics.jsonl file with summary metrics for each task and model configuration.

The requests directory contains all the exact model requests used for each instance.