TEMPO: A Realistic Multi-Domain Benchmark for Temporal Reasoning-Intensive Retrieval

TEMPO is the first benchmark combining temporal reasoning with reasoning-intensive retrieval across 13 diverse technical domains. Unlike existing benchmarks that focus on simple fact-seeking queries (e.g., "When did X happen?"), TEMPO targets complex, real-world information needs that require synthesizing evidence across time periods, tracking evolution, and comparing historical baselines with current states.

Table of Contents

• Dataset Summary
• Dataset Structure
  • Configurations
  • Data Instances
  • Data Fields
• Statistics
• Domains
• Key Features
• Citation

---

Dataset Summary

Real-world information needs often require reasoning about temporal evolution and synthesizing evidence across time periods. However, existing benchmarks either focus on simple date-lookup (Temporal QA) or logical inference without temporal grounding (Reasoning-Intensive Retrieval).

TEMPO addresses this gap by providing:

1. 1,730 Complex Queries: Naturally occurring, expert-level questions from Stack Exchange requiring deep temporal reasoning.
2. 1.6M+ Documents: A realistic retrieval corpus containing positive evidence and "hard" temporal negatives.
3. Step-wise Retrieval Planning: 3,976 decomposed retrieval steps mapped to gold documents for multi-hop evaluation.
4. Multi-Level Annotation: Fine-grained annotations for temporal intent, reasoning classes (e.g., trend analysis, event localization), and key time anchors.

Paper: TEMPO: A Realistic Multi-Domain Benchmark for Temporal Reasoning-Intensive Retrieval Repository: GitHub Link

---

Dataset Structure

Configurations

The dataset is organized into four configurations (subsets) to support different retrieval and reasoning tasks. You can load specific configurations as follows:

from datasets import load_dataset

# Load the Corpus (Knowledge Base)
corpus = load_dataset("tempo26/Tempo", "documents")

# Load the Main Retrieval Task (Query -> Docs)
queries = load_dataset("tempo26/Tempo", "examples")

# Load the Multi-Step Task (Query -> Step -> Docs)
steps = load_dataset("tempo26/Tempo", "steps")

# Load Reasoning/CoT Augmented Queries
reasoning = load_dataset("tempo26/Tempo", "deepseek_reason")

Data Fields

1. documents (The Corpus)

Contains the searchable knowledge base.

• id: Unique document identifier (e.g., bitcoin/585a73c2.txt).
• content: The full text content of the document.

2. examples (Main Queries)

Standard retrieval queries with gold standards.

• id: Unique query identifier (e.g., 126019_1).
• query: The complex natural language question/post.
• gold_ids: List of document IDs that contain the answer.
• gold_answers: The reference answer text (HTML/Markdown formatted).
• negative_ids: Query-specific hard-negative document IDs excluded by the paper/leaderboard evaluation protocol before ranking. Keep these documents in the run to report the stricter full-corpus protocol instead.
• query_guidance: A dictionary containing rich temporal metadata:
  • temporal_intent: (e.g., when, duration, before_after).
  • temporal_reasoning_class_primary: The type of reasoning required (see below).
  • key_time_anchors: List of explicit time expressions.

3. steps (Step-wise Evaluation)

Used for evaluating multi-hop temporal reasoning (Task 2).

• id: Step identifier.
• query: The context of the original query.
• step_instruction: (If available) The specific sub-question for this step.
• gold_ids: The specific documents relevant only to this reasoning step.

4. deepseek_reason

Queries augmented with Chain-of-Thought (CoT) reasoning paths generated by DeepSeek-32B.

• id: Query identifier.
• query: The original query.
• reasoning: The generated CoT reasoning trace (inside <think> tags) followed by the final reformulated answer/plan.

---

Statistics

---

Domains

The dataset covers 13 expert domains from Stack Exchange, split into 4 categories:

1. Blockchain: Bitcoin, Cardano, IOTA, Monero.
2. Social Sciences: Economics, Law, Politics, History (Largest domain).
3. Applied Fields: Quantitative Finance (Quant), Travel, Workplace, Genealogy.
4. STEM: History of Science and Mathematics (HSM).

This diverse coverage ensures that models are tested on specialized vocabulary and domain-specific temporal logic (e.g., block heights in Bitcoin vs. legislation enactment dates in Law).

---

Key Features

Temporal Reasoning Classes

Queries are annotated with 10 fine-grained reasoning classes:

• Event Analysis & Localization (EAL): Pinpointing when events occurred.
• Time Period Contextualization (TPC): Situating phenomena in historical periods.
• Origins & Evolution Comparative (OEC): Tracking concept evolution.
• Trends & Cross-Period Comparison (TCP): Comparing states across eras.
• (Plus: Event Verification, Causation Analysis, Materials Provenance, etc.)

Hard Negatives

To prevent simple lexical matching, TEMPO includes "Hard Negatives" (mined via GPT-4o). These documents are topically relevant (sharing keywords and entities) but are temporally mismatched (e.g., discussing the wrong year, an outdated version of a protocol, or a different legislative era).

---

Citation

If you use TEMPO in your research, please cite our paper:

soon

Note: This dataset is derived from public Stack Exchange data (CC-BY-SA).