LMCMark: A Human-Annotated Bilingual Citation Marker Dataset

LMCMark is a rigorous, human-annotated bilingual dataset constructed from verifiable news sources and academic paper corpus for the FreeCite benchmark. It contains over 40,000 validated citation markers across 5,858 query-response pairs spanning 21 diverse topics.

1. Data Construction

The construction of LMCMark follows a three-stage pipeline:

1. Source Collection: Reference documents are collected from two sources: Xinhua News Agency (bilingual news coverage) and REASONS (academic corpus from ArXiv, 2017–2024).
2. Response Generation: The Xinyu Online framework is employed to generate query-response pairs. The queries follow diverse formats aligned with established instruction-tuning datasets (including COIG, Belle, and PCLUE). For each query, the system retrieves documents from the collected corpus and generates a candidate response containing preliminary citation markers.
3. Granular Annotation: 27 expert raters (all STEM knowledge workers holding at least a Master's degree) verify, correct, and refine the citations over an intensive 45-day annotation campaign. Each instance is independently annotated by three raters, and final labels are determined by majority vote.

Annotation Criteria

To adjudge a citation as "Correct," the following rubric is applied:

• Contextual Relevance: A citation is valid if the generated sentence is supported by the reference material, and key concepts can be explicitly linked back to the source. Strict full-sentence entailment is not required, provided the core information is substantiated.
• Substantiality: The supported content must represent a reasonable proportion of the sentence's claim.

Quality Assurance

• Calibration Phase: Before mass annotation, all annotators processed a calibration batch reviewed by senior authors.
• Random Spot-Checks: Post-annotation, multiple rounds of random sampling were audited by senior researchers. Failed batches were reassigned for full re-annotation.

2. Dataset Statistics

The final LMCMark corpus spans 21 diverse topics, comprising 5,858 query-response pairs and over 40,418 verified citation markers, covering both English (10,000 citations) and Chinese (30,418 citations). The dataset has an average density of 6.90 citations per response.

Citation Count by Topic

Total: 5,858 query-response pairs, 40,418 citations

Data Structure

Each entry in LMCMark is stored as a triplet: (q, D, R*), where R* is the ground-truth response containing embedded citation markers (e.g., "...climate control [1] [3]."). By decomposing R* into segments, multiple "test instances" can be derived from a single response, enabling evaluation of the model's precision at every exact citation point.

Project Directory Structure Overview

This document explains the structure and contents of the project directories.

Ablation_data

The Ablation_data directory contains the experimental data and statistical results from the ablation studies. The subdirectories within it correspond to different variations of the ablation experiments:

• ref_deleted: Data for experiments where references are removed (ablation of references).
• ref_order: Data for experiments testing the effect of reference order (whether the references are provided in a specific order).
• ref_reverse_order: Data for experiments where references are provided in reverse order.
• single_sentence: Data for experiments involving a single sentence as input, to assess its impact.

Leaderboard_data

The Leaderboard_data directory contains the main experimental data related to the leaderboard models and their citation capabilities. The subdirectories are:

• confidence_interval: Data for generating confidence interval, used to assess the statistical significance and variability of the experimental results.
• main_data: Data on the experiments conducted with 20 models, evaluating their ability to handle citations in a concatenated prompt setup.
• main_stats: Statistical results from the above experiments.
• manual_label: Data from experiments where manual labeling was used to evaluate the models' performance.
• post_mounted_format: Results from experiments where models were labeled in a post-mounted format.
• xml_format: Results from experiments using the XML format for model outputs.

ALCE_data

The ALCE_data directory contains the main experimental data related to citation ability in ALCE papers. The subdirectories are as follows:

• 100data: 16 models were used to conduct 100 sample experiments to evaluate their ability to handle citations in the evaluation method used by ALCE.
• 100sample: 100 sample data were formed by extracting 20 samples from the original data provided by ALCE articles.
• data: Original data provided by ALCE article.
• result: Original experimental data using 16 models to evaluate their ability to handle citations in the ALCE evaluation method.

LongCite_data

The LongCite_data directory contains the main experimental data related to the citation power of LongCite papers. The subdirectories are as follows:

• preds: 16 models were used to conduct 100 sample experiments to evaluate their ability to handle citations in the evaluation method used by LongCite.
• scores_cite: Scoring results of 100 sample data.
• 100sample.json: 100 sample data randomly extracted from the original data provided by LongCite article.
• LongBench-Cite.json: Original data provided by LongCite article.

According to the directory introduction, you can select the experimental data directory you want to view and download.

Dataset Loading Instructions

This document explains how to load different subsets of the dataset using the Hugging Face datasets library.

Loading the Dataset

You can load the data from different parts of the dataset by specifying the correct data_dir and dataset split.

from datasets import load_dataset

# Load the whole Ablation_data dataset
ablation_data = load_dataset('flozxwer/LMCMark', data_dir='Ablation_data')

# Load a specific subdirectory within Ablation_data, e.g., ref_deleted
ablation_ref_deleted = load_dataset('flozxwer/LMCMark', data_dir='Ablation_data/ref_deleted')

# Load the entire Leaderboard_data dataset
leaderboard_data = load_dataset('flozxwer/LMCMark', data_dir='Leaderboard_data')

# Load a specific subdirectory within Leaderboard_data, e.g., main_data
leaderboard_main_data = load_dataset('flozxwer/LMCMark', data_dir='Leaderboard_data/main_data')

Ensure that you have the datasets library installed and that your environment has access to the dataset repository. If you encounter issues, make sure the directory paths match your dataset structure.

Citation

The data sources for the prediction results of each model are as follows:

• ALCE experiment: Downloaded directly from the data provided in the original link: GitHub - Princeton-NLP/ALCE

• LongCite experiment: Downloaded the experimental data from the data provided in the original link: GitHub - THUDM/LongCite, and randomly sampled 100 data to form the final experimental data.

License

The LMCMark dataset and its associated materials are released under the Apache License 2.0.

Contact

The original author hosts the code for applying this dataset on https://github.com/flozxwer/FreeCite