Tiny Theory of Mind

Tiny Theory of Mind is our first attempt at evaluating theory of mind capabilities in small language models. The benchmark covers a wide variety of ToM topics, ranging in difficulties that, for humans, would be appropriate for Pre-K through 6th grade.

The benchmark is designed primarily for base-model continuation log-likelihood scoring. It does not require instruction following, chain-of-thought, or generated explanations. Random-choice accuracy is 25%.

At a glance

Baseline Results

View all 36 baseline results

Rank	Model	Score
1	HuggingFaceTB/SmolLM-135M	47.20%
2	BananaMind/BananaMind-2-Pro	46.65%
3	HuggingFaceTB/SmolLM2-135M	46.55%
4	AxiomicLabs/GPT-X2-125M	44.55%
5	AxiomicLabs/GPT-X2.5-135M	44.20%
6	SupraLabs/Supra2-100M-Base	43.00%
7	AxiomicLabs/GPT-X-125M	42.75%
8	facebook/opt-125m	42.75%
9	SupraLabs/Supra-50M-Base	41.90%
10	BananaMind/BananaMind-2-Medium	41.20%
11	BananaMind/BananaMind-2-Medium-Chat	40.95%
12	TobiasLogic/Museko-125M	40.90%
13	openai-community/gpt2	40.75%
14	SupraLabs/Supra1.5-50M-Base-exp	39.95%
15	finnianx/Gros-Michel-90m-Base-v2	39.80%
16	BananaMind/BananaMind-2-Mini	38.55%
17	veyra-ai/Veyra2-Apricot-50M-Base	38.20%
18	BananaMind/BananaMind-2.1-Unified	38.15%
19	GODELEV/Archaea-74M	37.15%
20	BananaMind/BananaMind-2-Nano	37.10%
21	finnianx/michel-micro	36.40%
22	fromziro/Er-Large-30M	35.50%
23	EleutherAI/pythia-31m	35.20%
24	BananaMind/BananaMind-2-MoE	34.05%
25	BananaMind/BananaMind-2-SLMoE	34.00%
26	AxiomicLabs/GPT-S2-5M	32.65%
27	User01110/CMA-8M	32.55%
28	AxiomicLabs/GPT-S-5M	31.60%
29	BananaMind/BananaMind-2-Micro	31.60%
30	User01110/CMA-1M-Mini	31.40%
31	fromziro/Syn-2.6M	30.30%
32	AtomixLabs/Photon-2.0-1M	30.05%
33	AxiomicLabs/GPT-S-1.4M	29.95%
34	BananaMind/BananaMind-2.1-Pico-Preview	29.20%
35	ThingAI/Quark-50m-v2	29.10%
36	harley-ml/dillion-1.2M	28.30%

Task Format

Every item contains an unfinished Theory of Mind scenario in ctx, four possible continuations in endings, and a zero-based correct-answer index in label.

{
  "ind": 1910,
  "activity_label": "tiny_theory_of_mind_natural_continuation::first_order_false_belief::order_1::medium",
  "ctx_a": "Before entering a concert, Rae leaves her bicycle at the east rack. While she is inside with no view outdoors, staff move it to the west rack for construction. Until Rae receives new information, she will believe her bicycle is at",
  "ctx_b": "",
  "ctx": "Before entering a concert, Rae leaves her bicycle at the east rack. While she is inside with no view outdoors, staff move it to the west rack for construction. Until Rae receives new information, she will believe her bicycle is at",
  "endings": [
    " the east bicycle rack.",
    " the west bicycle rack.",
    " the construction office.",
    " the concert entrance."
  ],
  "source_id": "tiny_theory_of_mind_batch_008_natural_continuation_1910",
  "split": "test",
  "split_type": "synthetic",
  "label": "0",
  "metadata": {
    "answer": "the east bicycle rack",
    "unit": "text continuation",
    "topic": "first_order_false_belief",
    "difficulty": "medium",
    "grade_band": "2-4",
    "template_id": "theory_of_mind_continuation",
    "format_note": "Continuation-style prompt for base-model length-normalized log-likelihood scoring; endings include leading spaces.",
    "setting": "concert_bicycle_east_west_rack_move",
    "vocabulary_themes": [
      "transport",
      "false_belief"
    ]
  }
}

The correct continuation in this example is option 0:

Before entering a concert, Rae leaves her bicycle at the east rack. While she is inside with no view outdoors, staff move it to the west rack for construction. Until Rae receives new information, she will believe her bicycle is at the east bicycle rack.

Dataset composition

Theory of Mind Constructs

Each of 40 constructs has 50 examples (2.5% of the dataset), split evenly between easy and medium. Each cell shows the number of examples in that grade band.

Total examples: 2,000.

A note on difficulty: Although the benchmark is split 50/50 between easy and medium difficulty samples, individual theory of mind constructs are difficult to balance by grade due to differing ages when human children acquire theory of mind concepts. Some concepts are naturally acquired later. As such, you'll see some constructs, like pluralistic ignorance, weigh more heavily towards older grades, whereas diverse desires leans towards younger grades. Overall, an attempt has been made to balance by overall difficulty within each construct, and thus within the overall benchmark, rather than balancing by grade level.

Difficulty

Note: Difficulties are marked, like grade levels, as references to human standards. These may not, and often do not, reflect the general difficulty language models have answering.

Grade bands

Correct answer positions

The labels are exactly balanced, preventing answer-position frequency from providing an advantage.

License

Tiny Theory of Mind (Tiny ToM) is released under the Apache License 2.0.

Acknowledgments

Special thanks to the following people who made Tiny Theory of Mind possible:

Mmorgan-ML - Project Lead

Datdanboi - Invaluable advice & guidance on making benchmarks

Amanda Long - Brainstorming and help creating 60 early draft questions

The AxiomicLabs team for their constant encouragement.