WikiDYK

Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection

     

WikiDYK is the official implementation for Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection, accepted to ACL 2026 (Main Conference).

WikiDYK is a real-world, continuously extensible knowledge injection benchmark built from recently-added, human-written facts in Wikipedia's "Did You Know..." (DYK) entries. The benchmark contains 12,290 facts and 77,180 questions spanning formats from cloze prompts to complex multi-hop questions. Our experiments show that causal language models (CLMs) can be significantly weaker knowledge memorizers than bidirectional language models (BiLMs) after continued pre-training, and that ensembles of BiLMs can serve as external knowledge repositories for LLMs.

Proposed knowledge injection evaluation workflow. Knowledge from daily-updated DYK pages is injected via continued pre-training with various architectures and objectives. The injected model is then evaluated with questions spanning easy cloze prompts to complex multi-hop questions.

News

• [2026.06.02] WikiDYK has been accepted to ACL 2026.
• [2025.05.18] Paper released on arXiv.
• [2025.05.16] Dataset released on Hugging Face.

Table of Contents

• Overview
• Dataset
• Results
• Installation
• Data Preparation
• Run Examples
  • Training
  • Evaluation
  • Ensemble Model
• Citation

Overview

Example of the Wikipedia "Did You Know..." page that serves as the data source for WikiDYK.

WikiDYK leverages Wikipedia's "Did You Know..." pages, which highlight daily updates of facts reviewed by expert Wikipedia editors. Each day, about 10 facts are selected from recently created or expanded articles, ensuring both novelty and quality. We collect facts from January 2022 to April 2025, yielding 12,290 entries that are unlikely to exist in LLM pre-training data.

The code in this repository supports:

• Continued pre-training with next-token prediction (NTP), synthetic QA, and span prediction (SP) objectives.
• Multi-dimensional evaluation covering reliability, generality, paraphrase, portability, and locality questions.
• BiLM ensemble pipeline that combines specialized bidirectional models as external knowledge repositories integrated with LLMs via a scope classifier.

Dataset

Key characteristics:

• Real-world facts: collected from recently-added Wikipedia "Did You Know..." entries (Jan 2022 -- Apr 2025).
• Editor-curated source: based on entries selected by Wikipedia editors for verifiability and clarity.
• Multi-dimensional QA: evaluates knowledge memorization (reliability, generality, paraphrase), knowledge association (portability), and catastrophic forgetting (locality).
• Continuously extensible: future Wikipedia updates can naturally extend the benchmark with ~10 new facts daily.

Results

Evaluation results of injecting 1,000 facts from WikiDYK. BiLMs (Flan-T5) significantly outperform CLMs (Llama, Gemma) of comparable or larger size under aligned training objectives.

Key findings:

• NTP is not suitable for knowledge injection -- match accuracy remains below 1% for most question types after NTP training on CLMs.
• BiLMs are much more effective -- Flan-T5-770M achieves 46% reliability match accuracy (vs. <17% for the best CLM) despite being much smaller.
• Architecture matters more than objective -- comparing span prediction on both CLMs and BiLMs shows the advantage comes from bidirectional attention, not just the training objective.
• Ensemble pipeline further improves performance -- combining 10 specialized Flan-T5 models with a scope classifier yields an additional 6.7--29.1% improvement in reliability.

Effect of number of knowledge injected (upsampling s=1,000). BiLMs maintain stable performance as data scales up, while CLMs degrade sharply.

Effect of number of upsampling with 1,000 injected facts. Performance generally improves with more upsampling but saturates around s=6,000.

Installation

git clone git@github.com:zhang-yu-wei/WikiDYK.git
cd WikiDYK
conda create --name wikidyk python=3.10 -y
conda activate wikidyk
python -m pip install -r requirements.txt
python -m pip install flash-attn --no-build-isolation

Data Preparation

Download the ready-to-run experiment data from Hugging Face:

huggingface-cli download YWZBrandon/wikidyk_exp --repo-type dataset --local-dir data

Run Examples

Training

NTP Training

bash scripts/train_ar_full_all_8gpu.sh

QA Training

bash scripts/train_synthetic_qa_all_8gpu.sh

SP Training for CLMs

bash scripts/train_pred_mask_full_all_8gpu.sh

SP Training for BiLMs

bash scripts/train_full_flan_t5.sh

Evaluation

Static Analysis

bash scripts/eval_qa_static.sh

RAG

bash scripts/eval_qa_rag.sh

Evaluation on Trained CLMs

bash scripts/eval_qa_trained.sh

Evaluation on Trained BiLMs

bash scripts/eval_qa_accelerate.sh

Ensemble Model

Train the scope classifier:

bash scripts/train_scope_clf_deberta_large.sh

Train models on clustered data:

bash scripts/train_cluster_s10_flan_t5_base_us1000.sh
bash scripts/train_cluster_s10_flan_t5_large_us1000.sh

Then evaluate the trained BiLMs with scripts/eval_qa_accelerate.sh. To run the full ensemble pipeline, modify the ensemble configs in src/ens_configs/config_t5_base_s10.json and src/ens_configs/config_t5_large_s10.json, then run:

python src/ensemble_pipeline.py --ens_config src/ens_configs/config_t5_large_s10.json --no_factual

Citation

If you find WikiDYK useful in your research or applications, we would appreciate it if you could cite our work:

@misc{zhang2025bidirectionallmsbetterknowledge,
      title={Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection}, 
      author={Yuwei Zhang and Wenhao Yu and Shangbin Feng and Yifan Zhu and Letian Peng and Jayanth Srinivasa and Gaowen Liu and Jingbo Shang},
      year={2025},
      eprint={2505.12306},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.12306}, 
}