TEMPO: A Realistic Multi-Domain Benchmark for Temporal Reasoning-Intensive Retrieval

🚨 News

• [2026-01] 🛠️ Code: Full evaluation code and temporal metrics are released.

---

Existing temporal QA benchmarks focus on simple fact-seeking queries, while reasoning-intensive retrieval benchmarks lack temporal grounding. TEMPO bridges this gap as the first benchmark combining temporal reasoning with reasoning-intensive retrieval across 13 domains.

TEMPO Features:

1. 1,730 Complex Queries: Requiring deep temporal reasoning (tracking changes, trends, cross-period evidence).
2. Step-wise Retrieval Planning: 3,976 decomposed steps with gold documents mapped for multi-hop evaluation.
3. Novel Temporal Metrics: Introducing Temporal Coverage@k and Temporal Precision@k to measure temporal completeness.

---

🏆 Leaderboard

---

⚙️ Setup & Usage

1. Installation

Clone the repository and install the dependencies:

git clone https://github.com/tempo-bench/Tempo.git
cd Tempo
pip install -r requirements.txt

2. Retrieval Evaluation

Run standard retrieval evaluation using run.py. This supports running over all domains or specific ones.

# Evaluate retrieval on the 'history' domain with BM25
python run.py --task history --model bm25 --output_dir outputs

# Evaluate on ALL domains
python run.py --task all --model bm25 --output_dir outputs

3. Step-wise Evaluation

For deeper analysis, evaluating the intermediate retrieval steps:

python run_step.py --task economics --model e5 --output_dir outputs_steps

4. 📈 Calculating Temporal Metrics (New!)

We provide a dedicated script temporal_metrics.py to calculate advanced temporal metrics like Temporal Coverage and Temporal Precision. This script uses an LLM to judge the temporal relevance of retrieved documents.

Note: You must perform step 1 first to install the required provider libraries (e.g., openai, anthropic, google-generativeai).

The script automatically downloads the necessary queries and corpus data from the Hugging Face Hub (dataset tempo26/Tempo), so you don't need to manually download data files.

Supported Providers & Configuration

Set the environment variables for your chosen provider:

🔷 Azure OpenAI (Default)

export AZURE_OPENAI_ENDPOINT="https://your-endpoint.openai.azure.com/"
export AZURE_OPENAI_API_KEY="your-api-key"
export AZURE_DEPLOYMENT_NAME="gpt-4o"

🟢 OpenAI

export OPENAI_API_KEY="sk-..."
export OPENAI_MODEL="gpt-4o" # Optional

🟠 Anthropic (Claude)

export ANTHROPIC_API_KEY="sk-ant-..."
export ANTHROPIC_MODEL="claude-3-5-sonnet-20240620" # Optional

🔵 Google (Gemini)

export GOOGLE_API_KEY="AIza..."
export GEMINI_MODEL="gemini-1.5-pro" # Optional

Running the Script

Run the script specifying the provider using --provider.

# Using Azure (default)
python temporal_metrics.py \
    --model_results_dir ./outputs \
    --k_values 10 20 \
    --provider azure

# Using OpenAI
python temporal_metrics.py --model_results_dir ./outputs --k_values 10 --provider openai

# Using Claude
python temporal_metrics.py --model_results_dir ./outputs --k_values 10 --provider anthropic

---

📊 Benchmark Comparison

📝 Citation

If you use TEMPO in your work, please cite our paper:

@article{abdallah2026tempo,
  title={TEMPO: A Realistic Multi-Domain Benchmark for Temporal Reasoning-Intensive Retrieval},
  author={Abdallah, Abdelrahman and Ali, Mohammed and Abdul-Mageed, Muhammad and Jatowt, Adam},
  journal={arXiv preprint arXiv:2601.09523},
  year={2026}
}