ECT-QA: Earnings Call Transcripts for Temporal Question Answering

ECT-QA is a comprehensive benchmark dataset designed for evaluating time-sensitive question answering capabilities in RAG systems. The dataset is built from real-world Earnings Call Transcripts (ECTs) and features complex temporal reasoning questions that require understanding evolving financial knowledge.

🔥 This dataset is one part of our paper: "RAG Meets Temporal Graphs: Time-Sensitive Modeling and Retrieval for Evolving Knowledge" (https://arxiv.org/abs/2510.13590)

📊 Dataset Overview

• Source: 480 Earnings Call Transcripts from 24 publicly traded companies (2020-2024)
• Total Queries: 1,105 specific questions + 100 abstract questions
• Corpus Size: 1.58 million tokens
• Temporal Coverage: 2020-2024 with incremental update support

🎯 Key Features

Temporal Question Types

• Specific Multi-hop Questions: Fine-grained factual queries requiring temporal reasoning
  • Single-time queries (within one time point)
  • Multi-time queries (across multiple periods)
  • Relative-time queries (before/after specific references)
• Abstract Questions: High-level analytical queries requiring summarization and trend analysis

Incremental Evaluation Design

• Base Corpus: 2020-2023 documents (384 transcripts)
• New Corpus: 2024 documents (96 transcripts)
• Base Queries: Answers rely on 2020-2023 knowledge
• New Queries: Answers require 2020-2024 knowledge

📁 Dataset Structure

ECT-QA/
├── data/
│ ├── old/ # 2020-2023 documents (JSON files per company/quarter)
│ └── new/ # 2024 documents (JSON files per company/quarter)
└── questions/
│ ├── local_questions_old.json
│ ├── local_questions_new.json 
│ ├── global_questions_old.json 
│ └── global_questions_new.json 

📥 Loading the Dataset

The dataset can be loaded using the Hugging Face datasets library:

from datasets import load_dataset

# Load questions dataset
questions = load_dataset("austinmyc/ECT-QA", "questions")

# Load corpus dataset
corpus = load_dataset("austinmyc/ECT-QA", "corpus")

# Or load default (questions)
dataset = load_dataset("austinmyc/ECT-QA")