OMI-Bench: Olympiad-level Multi-Image Science Benchmark

OMI-Bench is a multimodal, multi-image benchmark of olympiad-level science questions across Biology, Chemistry, Mathematics, and Physics. Every question is grounded in two or more images (diagrams, figures, structures, experimental setups), making it a test of genuine multi-image scientific reasoning rather than single-figure perception.

Dataset at a glance

Schema

Each line in test.jsonl is one question:

Image placeholder convention

[IMAGEn] is a global 0-based index into image_list. Tags in the question come first; for image-answer questions the option tags continue the numbering. The union of all [IMAGEn] tags in a record covers exactly its image_list.

Usage

import json
from PIL import Image

rows = [json.loads(l) for l in open("test.jsonl")]
r = rows[0]
imgs = [Image.open(f"images/{name}") for name in r["image_list"]]
# render r["question"], substituting [IMAGE0..n] with imgs

⚠️ Contamination note: solution contains the full reasoning and the answer. For clean evaluation, prompt models with question (+ choice_list) only and score against answer; do not feed solution.

Provenance & licensing

Items are aggregated from established multi-image benchmarks (ReMI, SeePhys) and science-olympiad / exam materials, plus a small number of CC-licensed third-party figures. Per-source attribution and licensing obligations are documented in SOURCES.md.

The dataset card is released under CC BY 4.0; individual source items retain their upstream licenses and citation requirements (see SOURCES.md). If you use OMI-Bench, please also cite the upstream ReMI and SeePhys benchmarks.

Known limitations

• Answers are inherited from the source materials and have not been independently re-verified for scientific correctness. Structure, answerability, and image–question linkage were checked; the truth of each gold answer was not re-derived. Treat answer keys as source-provided.
• solution chains are model-/annotator-generated reasoning, not peer-reviewed proofs; they may contain errors even when the final answer is correct.
• Some open-ended answers are free-form strings (e.g. "a) FALSE b) TRUE …"), requiring answer-normalization for automatic scoring.
• 11 records carry lettered options inline (see has_inline_choices); a few physics answers encode tables as flattened text.
• English only.

Changelog (release preparation)

• Added explicit answer_type field to every record.
• Removed an internal annotation field.
• Repaired 2 malformed multiple-choice records.
• Trimmed per-question image_lists to referenced images and re-indexed tags.
• Restored 9 legacy physics records from the older physics JSON export after cleaning formatting artifacts and raw image placeholders (restored_records.csv).
• Replaced 4 CC BY-NC-ND (NonCommercial/NoDerivatives) photos in biology-53/54/55/105 with newly generated, license-clean illustrations (gpt-image-2); records keep their multi-image layout. No questions dropped.
• Cleaned text-hygiene issues (stray control chars, leaked JSON fragments, legacy <image_n> tags, unpaired $); tagged 11 inline-choice records.