---

📰 News

[!NOTE] Next up: registration, the participant mailing list, and the competition platform. They will be announced here and on the participation guide.

---

⚡ Quick start

# 1 — get the data (v1.1, ~3.2 GB; everything you need, in one download)
pip install huggingface_hub
hf download DataScience-UIBK/RETECO-SemEval2027 --repo-type dataset --local-dir reteco_data

# 2 — get the baselines
git clone https://github.com/DataScienceUIBK/RETECO.git
cd RETECO/starter_kit && pip install -r requirements.txt

# 3 — develop on train, then check yourself on dev
python official_baseline.py --data ../../reteco_data --splits train
python official_baseline.py --data ../../reteco_data --splits dev

New to the task? The participation guide walks through choosing a sub-track, building a system, and scoring it locally with the same metric the leaderboard uses.

🎯 Tracks

See the full task definitions and evaluation plan.

📊 Training and development data

The official train/dev release is on Hugging Face: DataScience-UIBK/RETECO-SemEval2027

hf download DataScience-UIBK/RETECO-SemEval2027 --repo-type dataset --local-dir reteco_data

One download contains everything needed for any sub-track: the full retrieval corpus of all 24 domains, task files, and TREC qrels for both splits.

The split is 70/30 per domain with a fixed seed. The corpus is never split; Track 1 splits on the query (steps are nested inside their query record) and Track 2 splits on the whole conversation. Gold judgments ship for both splits — the SemEval test set is separate and never released.

🧰 Starter kit

starter_kit/ has the baselines, scorer, format checker, and the build script that produces the release from pinned upstream revisions, plus the deduplication script behind data v1.1 and its verifier.

cd starter_kit && pip install -r requirements.txt
python official_baseline.py --data ../../reteco_data --splits train dev

Reference BM25, nDCG@10, macro-averaged over domains:

Track 1 numbers are on data v1.1. Per-domain numbers: starter_kit/BASELINE_RESULTS.md.

🔍 Sample data

The repository contains a compact, human-inspected package copied from the official TEMPO and RECOR releases:

• Track 1: 5 TEMPO examples, 26 supporting passages, and 26 positive qrels.
• Track 2: 4 RECOR conversations containing 13 turns, 15 supporting passages, and 32 positive qrels.
• A manifest pins the source dataset revisions and selected record IDs.

Inspect samples online · Download the ZIP · Browse source files

The sample is a format and feasibility demonstration—not a new benchmark split and not the hidden SemEval test set. It contains only referenced positive evidence, so retrieval systems must not be evaluated against the sample corpus alone.

🗂️ Data and corpora

Track 1: TEMPO

TEMPO contains 1,730 complex temporal queries, 3,976 decomposed retrieval steps, and 1,654,055 documents across 13 independent domain corpora. RETECO v1.1 keeps one copy of each duplicated text, leaving 1,167,159 documents.

• Dataset: tempo26/Tempo on Hugging Face
• Code and baselines: tempo-bench/Tempo
• Paper: arXiv:2601.09523

Track 2: RECOR

RECOR contains 707 conversations, 2,971 target turns, and 507,141 documents across 11 domain corpora. Six domains come from BRIGHT and five from StackExchange.

• Dataset: RECOR-Benchmark/RECOR on Hugging Face
• Code and baselines: RECOR-Benchmark/RECOR
• Published paper: ACL Anthology 2026.findings-acl.129

These are the upstream sources. The RETECO release above packages them into train/dev splits with qrels. Final SemEval scoring uses a separate, never-publicly-released test set with private gold judgments.

📄 Papers

TEMPO

TEMPO: A Realistic Multi-Domain Benchmark for Temporal Reasoning-Intensive Retrieval
Abdelrahman Abdallah, Mohammed Ali, Muhammad Abdul-Mageed, and Adam Jatowt.
arXiv:2601.09523, 2026. Paper

@article{abdallah2026tempo,
  title={TEMPO: A Realistic Multi-Domain Benchmark for Temporal Reasoning-Intensive Retrieval},
  author={Abdallah, Abdelrahman and Ali, Mohammed and Abdul-Mageed, Muhammad and Jatowt, Adam},
  journal={arXiv preprint arXiv:2601.09523},
  year={2026},
  url={https://arxiv.org/abs/2601.09523}
}

RECOR

RECOR: Reasoning-focused Multi-turn Conversational Retrieval Benchmark
Mohammed Ali, Abdelrahman Abdallah, Amit Agarwal, Hitesh Laxmichand Patel, and Adam Jatowt.
Findings of ACL 2026, pages 2688–2723. Paper · DOI

@inproceedings{ali2026recor,
  title={RECOR: Reasoning-focused Multi-turn Conversational Retrieval Benchmark},
  author={Ali, Mohammed and Abdallah, Abdelrahman and Agarwal, Amit and Patel, Hitesh Laxmichand and Jatowt, Adam},
  booktitle={Findings of the Association for Computational Linguistics: ACL 2026},
  pages={2688--2723},
  year={2026},
  publisher={Association for Computational Linguistics},
  doi={10.18653/v1/2026.findings-acl.129},
  url={https://aclanthology.org/2026.findings-acl.129/}
}

Both records are available in CITATIONS.bib. The official RETECO task-paper citation will be added after the SemEval-2027 task description is published.

🗺️ Repository map

RETECO/
├── docs/                         # GitHub Pages website
│   ├── index.html
│   ├── task.html
│   ├── data.html                 # Detailed per-domain corpora
│   ├── samples.html              # Human-readable examples
│   ├── evaluation.html
│   ├── papers.html               # Proposal, papers, citations
│   ├── participate.html
│   ├── timeline.html
│   ├── sample_data/              # Curated machine-readable sample
│   └── assets/
│       ├── downloads/
│       └── papers/
├── starter_kit/                  # Baselines, scorer, format checker, build script
├── .github/workflows/            # Pages deployment
├── CITATIONS.bib
├── CONTRIBUTING.md
├── CODE_OF_CONDUCT.md
├── SECURITY.md
└── README.md

Format checkers, local scorers, and starter baselines are in starter_kit/. Sample submissions will be added when the competition-platform interface is frozen.

💻 Local website preview

The website is dependency-free:

python -m http.server 8000 --directory docs

Then open http://localhost:8000/.

🌐 GitHub Pages deployment

The included workflow publishes docs/. In repository settings, choose GitHub Actions as the Pages source. The site URL is:

https://datascienceuibk.github.io/RETECO/

📅 Important dates

• Sample data: August 8, 2026
• Training data: released August 30, 2026
• Evaluation: January 10–31, 2027
• Paper and workshop dates: tentative; see the official SemEval-2027 calendar

👥 Organizers

Abdelrahman Abdallah, Mohammed Ali, Muhammad Abdul-Mageed, Kevin Duh, and Adam Jatowt.

For task questions, contact Abdelrahman Abdallah.

⚖️ License

The data release carries two licenses. Corpus and question/answer text is CC BY-SA 4.0 — all passage text originates from Stack Exchange, whose user contributions are licensed CC BY-SA, which is share-alike and cannot be relicensed. RETECO annotations (split assignments, TREC qrels, split_manifest.json) are CC BY 4.0. Attribution for the underlying content remains with the original Stack Exchange authors.

Existing benchmark data and code may use different licenses; consult the license attached to each resource. A repository-level software license will be added before the first tagged RETECO code release.