Ask-Before-Detection: Identifying and Mitigating Conformity Bias in LLM-Powered Error Detector for Math Word Problem Solutions

This is the implementation code for paper: Ask-Before-Detection: Identifying and Mitigating Conformity Bias in LLM-Powered Error Detector for Math Word Problem Solutions.

Dataset

The data used in this paper is located in the ./Data/ folder and consists of 200 questions sampled from the GSK8K test split. For each question, we provide two correct solutions—golden and reform—which differ in their problem-solving approaches.

To investigate how LLMs detect errors across these solutions, we introduce 4 types of errors in our experiments: calculation error, reference error, missing step, and irrelevant information (hallucination). Each solution is modified using two error-injection strategies: single-error cases (1,600 examples) and mixed-error cases (200 examples).

Experiment

The code for this paper is located in the ./Code folder and consists of two main steps. To run the experiment, first execute the data preparation script run_prepare.sh, which generates a reference solution for each provided solution. Then, run the main experiment script run_main.sh to obtain the final results.