📚 DetectiveQA: Evaluating Long-Context Reasoning on Detective Novels

中文版

📚 DetectiveQA Paper • 📊 DetectiveQA Dataset

🔥 Updates

🔥🔥🔥 [2025/1/13] We are very excited to release DetectiveQA, a dataset designed to evaluate large models' long-text narrative reasoning capabilities based on detective fiction. In this context, models will assume the role of detectives, performing reasoning based on long texts grounded in realistic scenarios! We hope this contributes to enhancing the long-text reasoning abilities of future Large Language Models (LLMs).

⚙️ How to evaluate on DetectiveQA

0. Prepare Data

Download the data from huggingface and save it in the /data folder.

1. Prepare Model Inference

• In models.py, follow the structure of other classes to implement your own model class. Implement the methods __init__, __answer__, and get_name:

  • __init__(self, q_len): Initialize the model. The q_len is the truncation length for the questions used in the answer method.
  • __answer__(self, question_list): This method returns the answer to the questions. The input is a list that contains a string question and a string prompt. The question contains:

    {novel content}

    Please answer the following question based on the provided content:

    {question}

    A: xxxx

    B: xxxx

    C: xxxx

    D: xxxx

    The prompt contains:

    Please return the result strictly in the format {"answer":"x","reasoning":"xxx"}, where the answer is your answer and reasoning is your reasoning process. Otherwise, the score cannot be calculated.

    The function's output should be a dictionary with two fields: one for answer (the answer to the question) and reasoning (the reasoning process). The output can either directly be in the dictionary format, which requires higher model capabilities (and you may need to use json.loads), or the model can first output the reasoning process, and then append the phrase "So the correct option is" to get the final output with the highest probability option from A, B, C, or D.

    { "answer": "x", "reasoning": "xxx" }

  • get_name(self): Return the name of the model to be displayed during evaluation.

• In get_model_name(model_name, model_config), implement the logic to extract the model's file name from the model_config. If no q_len parameter is provided, return the model name with the default truncation length.

2. Evaluation

Evaluation consists of both manual and automatic evaluation methods:

2.1 Automatic Evaluation

For automatic evaluation, you can use the script main_experiment.py. Modify the parameters in the script to run the evaluation. The script parameters are as follows:

    --model_config: The model configuration file
    --data_path: The path to the data files
    --save_path: The output path to save results
    --lang_list: The list of languages to evaluate (zh/en)
    --anno_part: The annotated part to evaluate (human_anno/AIsuo_anno)

2.2 Manual Evaluation

Obtain Model Output

Run the script model_evaluation.py to get the model's output. The parameters for this script are as follows:

    --data_path: The path to the data files
    --save_path: The output path to save results
    --lang: The language to evaluate (zh/en)
    --anno_part: The annotated part to evaluate (human_anno/AIsuo_anno)
    --config: The model configuration file

Here is an example command to run:

python model_evaluation.py --data_path ../data --save_path ./output/zh/ --lang zh --anno_part human_anno --config ./config/LWM.json

Output Alignment

Use answer_alignment.ipynb to align the model's output to the standard format. Alignment here refers to the process of ensuring the model's output follows the correct format. Some models may output unnecessary characters or incorrect answer formats (e.g., dictionaries), which need to be aligned to the standard {answer: xx, reasoning: xxx} format. This can be considered as a post-processing step.

Reasoning Evaluation

First, set the api-key at line 33 of BenchMark.py. Then, run get_judgement_from_gpt4.py to get the evaluation results from GPT-4. Note that some GPT-4 outputs might not conform to the required format. You can use judgement_purify.ipynb to clean up these results if necessary.

Evaluation Results

• 1. Use result_saving.ipynb to summarize and save the results for each data point.
• 2. Use get_model_final.ipynb to aggregate and display the model's evaluation results.

📝 Citation

@article{xu2024detectiveqa,
  title={Detectiveqa: Evaluating long-context reasoning on detective novels},
  author={Xu, Zhe and Ye, Jiasheng and Liu, Xiangyang and Sun, Tianxiang and Liu, Xiaoran and Guo, Qipeng and Li, Linlin and Liu, Qun and Huang, Xuanjing and Qiu, Xipeng},
  journal={arXiv preprint arXiv:2409.02465},
  year={2024}
}