Dataset Description

xNot360 is a dataset designed to evaluate the ability of large language models (LLMs) to handle logical negation. The "360" in the name reflects a comprehensive, all-encompassing approach to negation detection, emphasizing its importance across diverse contexts and sentence structures.

While LLMs can easily handle simple negations, such as using "not" or straightforward negation phrases, they often struggle with more complex structures. Logical negation is a challenging task, even for humans, which suggests that methods like RLHF (Reinforcement Learning from Human Feedback) may not be suitable. For this reason, xNot360 was created using classical logic as its foundation.

Key Characteristics

• Logical Framework: Adheres to classical logic principles to ensure consistency, including nuanced cases like conditional statements that often lead to errors.
• Balanced Dataset: Contains 360 samples, split equally into:
  • 180 positive samples (sentence pairs that logically negate each other).
  • 180 negative samples (sentence pairs that do not logically negate each other).
• Templates: Sentences are generated using diverse templates, with components logically negated.

Example Complexity

Consider the conditional statements:

• "If I study hard, I will pass the bar exam" (A → B or ¬A ∨ B).
• "If I do not study hard, I will not pass the bar exam" (¬A → ¬B or A ∨ ¬B).

Although these sentences may appear to negate each other, their logical forms differ. Such examples demonstrate the dataset's focus on identifying subtle inconsistencies in logical negation.

Dataset Details

• Language(s): English
• License: Apache-2.0
• Size: 360 sentence pairs

Dataset Sources

The dataset was created using sentence templates and applying logical principles to construct positive and negative pairs. Classical logic served as the primary guideline for generating the data.

Uses

Direct Use

• Evaluating LLMs on logical negation tasks.
• Training and fine-tuning models to enhance their understanding of logical reasoning.
• Analyzing performance on complex language understanding tasks.

Out-of-Scope Use

• Applications unrelated to logical negation detection.
• Tasks that do not emphasize logical reasoning accuracy.

Dataset Structure

The dataset consists of sentence pairs with labels:

• Positive Labels: Indicate sentence pairs logically negate each other.
• Negative Labels: Indicate sentence pairs do not logically negate each other.

Example Data Point

Bias, Risks, and Limitations

• The dataset focuses on logical negation and may not generalize to other linguistic phenomena.
• Templates may not encompass all real-world complexities.

Recommendations

Users should focus on the dataset's purpose: evaluating and improving LLMs' abilities in logical negation detection.

Citation

BibTeX:

@article{nguyen2023negation,
  title={A negation detection assessment of GPTs: analysis with the xNot360 dataset},
  author={Nguyen, Ha Thanh and Goebel, Randy and Toni, Francesca and Stathis, Kostas and Satoh, Ken},
  journal={arXiv preprint arXiv:2306.16638},
  year={2023}
}