DetectiveQA

This is a bilingual dataset with an average question length of 100K, containing a series of detective novel questions and answers. These questions and answers are extracted from detective novels and cover various types of questions, such as: character relationships, event order, causes of events, etc.

1. Data Source/Collection

The novels in the dataset come from a collection of classical detective novels we gathered. These novels have the following characteristics:

1. The novels have a clear sequence of events.
2. The novels have clear character relationships.
3. The novels have clear causes for events, with reasoning clues appearing before the answers.

We have two data annotation methods:

1. Manual Annotation: Annotators are asked to select relatively complex reasoning questions from the novels, provide answers to those questions, and offer a reasoning process for the answers. The reasoning process must include clues, the location of those clues, and a step-by-step explanation of how the clues lead to the answer.
2. AI-assisted Annotation: The annotation process is similar to manual annotation, but we use a closed-source AI model to assist in generating relevant content for the annotators' reference. The AI model extracts reasoning paragraphs from the novels and organizes them into multiple-choice questions. Annotators then use this reference information to label the data and derive the final annotations.

2. Dataset Composition

The dataset in the data folder contains four files: anno_data_zh, novel_data_zh, anno_data_en, and novel_data_en. The files anno_data_zh and anno_data_en contain the annotated data, while novel_data_zh and novel_data_en contain the raw novel data. The "zh" refers to the Chinese language, and the "en" refers to the English language.

• Novel Data (novel_data): The novel data consists of text files for each novel. Each novel file is named {novel_id}-{novel_name}-{author}.txt, where the content of each file corresponds to the novel. Each paragraph in the novel is numbered as follows:

[1] The Tenant of Room 13
[2] In Y-town (of course, in Tokyo), there was a building called the Kanto Building, which was not very large. Recently, the building had been put up for lease. One morning, a distinguished gentleman walked into the office of the building, and the receptionist took his business card. The card read "Art Dealer Hidetomo Inagaki."
[3] Inagaki, with a stout cane and a silver chain hanging from his white vest, arrogantly said:
[4] "If there’s an available room, I’d like to rent one."
...

• Annotated Data (anno_data): The annotated data consists of two folders: human_anno (manual annotations) and AIsup_anno (AI-assisted annotations). Each novel’s annotation is stored as a JSON file named {novel_id}.json. The JSON file contains the novel ID, the number of paragraphs, time spent, and a list of questions. The annotation format for each question is as follows:

{
    "question": "What is the relationship between A and B?",
    "options": {
        "A": "Option A",
        "B": "Option B",
        "C": "Option C",
        "D": "Option D"
    },
    "answer": "Answer (A/B/C/D)",
    "distraction": {
        "A": "Distraction reason for A",
        "C": "Distraction reason for C",
        "D": "Distraction reason for D"
    },
    "reasoning": [
        "Clue 1",
        "Clue 2",
        "Clue 3",
        "Reasoning process"
    ],
    "clue_position": [
        "Clue 1's paragraph number",
        "Clue 2's paragraph number",
        "Clue 3's paragraph number",
        -1
    ],
    "answer_position": "Answer's paragraph number"
}

3. Input Modes

Our dataset has four input modes:

1. simple: Only the question is provided, along with the title and author of the novel.
2. detailed: Long context plus question, where the novel content up to the answer paragraph is provided along with the question. If there is a length limitation, the context is truncated from the tail.
3. with_clue: Clues plus question, where the annotated clues and the question are provided.
4. only_question: This version only includes the question without options and is not used for final evaluation.

4. Evaluation Metrics

We use two evaluation metrics:

1. Question Accuracy: The accuracy of the questions is calculated as the proportion of questions that the model answers correctly out of the total number of questions.
2. Reasoning Process Effectiveness: This measures the effectiveness of the reasoning process output by the model. Specifically, it calculates the ratio of the number of clues mentioned in the reasoning process to the total number of clues. The number of clues mentioned is evaluated by ChatGPT-4. (The reliability of GPT-4 has been verified through manual annotation of 100 samples, with a Kappa coefficient and accuracy both reaching 92%, showing high reliability.)