ChroKnowBench

ChroKnowBench is a benchmark dataset designed to evaluate the performance of language models on temporal knowledge across multiple domains. The dataset consists of both time-variant and time-invariant knowledge, providing a comprehensive assessment for understanding knowledge evolution and constancy over time. Dataset is introduced by Park et al. in ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains

Dataset Overview

Domains and Characteristics

ChroKnowBench includes datasets from several domains with distinct characteristics:

• Time-variant Knowledge: Datasets containing facts that change over time, with two temporal states:

  • Dynamic: Knowledge that has evolved over an accumulated period. The number in parentheses represents the average change in objects per element within a dynamic dataset.
  • Static: Knowledge that did not change during the accumulated period, though it has potential to be changed in the future.

• Time-invariant Knowledge: Datasets that contain facts which remain constant, such as commonsense and mathematics.

* We've expanded the time frame of Biomedical ChroKnowBench, from 2020-2024 to 2010-2024. If you wish to use this version, download the version 2.
** For LEGAL dataset in time variant, it is the number of category like `Organization', as it is unstructured dataset without specific short relations.

Dataset Format

• Structured Format: The datasets use the format (subject, relation, object, [time]), where:
  • (s, r, o, t): Represents time-variant knowledge, where t is the temporal information.
  • (s, r, o): Represents time-invariant knowledge, without any temporal component.
• QA Format: Legal domain data is provided in a QA format to accommodate unstructured knowledge representation.

Statistics of Object Changes

Key Statistics:

• General Domain: Average changes per object: 2.6 (Moderate skewness)
• Biomedical Domain: Average changes per object: 8.9 (Balanced distribution)
• Legal Domain: Average changes per object: 1.1 (Highly skewed, minimal changes)

Observations:

• The Biomedical domain exhibits the most dynamic changes, showing a nearly balanced cumulative distribution.
• The General domain presents moderate skewness, with a broader range of changes.
• The Legal domain is highly skewed, with most changes concentrated in a single occurrence.

Dataset Download

For Biomedical dataset(Dynamic, Static and Fewshot), version 2 is released here. The first version of Biomedical dataset used in ICLR 2025 publication, is here. Please append each jsonl file to appropriate directory(/ChroKnowBench for Dynamic, Static file, and /ChroKnowBench/Fewshots for Fewshot file) Other benchmarks can be downloaded in this dataset repository.

git clone https://huggingface.co/datasets/dmis-lab/ChroKnowBench

Please download the dataset in Huggingface first, then download Biomedical datset from the link above.
📌 We currently do not support load_datset module. Please wait for an update.

Citation

If you use ChroKnowBench in your research, please cite our paper:

@inproceedings{park2025chroknowledge,
   title={ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains},
   author={Yein Park and Chanwoong Yoon and Jungwoo Park and Donghyeon Lee and Minbyul Jeong and Jaewoo Kang},
   booktitle={The Thirteenth International Conference on Learning Representations},
   year={2025},
   url={https://openreview.net/forum?id=whaO3482bs}
}

Contact

For any questions or issues, feel free to reach out to [522yein (at) korea.ac.kr].