FLAWS: Faults Localization Across Writing in Science

FLAWS is a benchmark for evaluating error identification and localization in scientific papers. It currently consists of 713 paper–error examples, including:

• 265 unique papers with one error inserted using GPT-5 (in ALL_OPENAI.tar.gz)
• 448 unique papers with one error inserted using Gemini 2.5 Pro (in ALL_GEMINI.tar.gz)

The dataset is generated using a systematic, autonomous framework that produces paper–error examples and evaluates them using a scalable automated metric.

The code and framework used to create this dataset can be found at this Github Repository.

(Note: The non-ml zip folders contain an additional extention to the main benchmark on CS papers instead of ML papers. Those datasets consists of 67 Gemini 2.5 Pro errors and 48 GPT-5 errors.)

---

Directory Structure

After extracting either zip file, you will see the following structure:

Folder Descriptions

altered_papers/

Contains folders with the LaTeX source of each paper and its modified version.

generated_claims/

Contains .txt files listing the claims generated for each paper.

inserted_error/

Contains .txt files with the error inserted into a specific claim in each paper.

location_error/

Contains .txt files with the text passages corresponding to each inserted error.

identified_errors/

Contains .txt files produced during the internal error-identification filtering step.

evaluation_errors/

Contains .txt files with error-identification outputs generated by different LLMs.

---

Metadata Files

*_all.json

A dictionary of all paper–error examples.