Temporal Knowledge Graph QA (tkgqa)

Answer time-dependent questions over a temporal knowledge graph — "Who was the president of Northland in 2005?", "Who was the first CEO of Fairview?", "When was Bob the mayor of Eastmark?" — by retrieving the relevant timed facts, reasoning over their time intervals, and abstaining when no time-consistent fact exists, with the supporting facts shown for provenance.

NLP in Industry — Final Assignment, Project #18. Author: Le Dinh Minh Quan (Student 23127460).

tkgqa runs a trainable dense fact retriever (BAAI/bge-small-en-v1.5, fine-tuned) behind a deterministic agent (D1–D5) that parses the temporal constraint, retrieves candidate timed facts, applies an explicit temporal operator (point-in-time / before-after / first-last / when), selects the answer, and abstains when uncertain. Only the retriever is trained; the temporal reasoning and abstention are algorithmic.

question ──► parse constraint ──► retrieve timed facts (dense + BM25 → RRF)
                                          │
   answer + supporting facts ◄── abstain? ◄── select ◄── temporal operator (point/before/first/when)

---

How this repo meets each assignment requirement

---

Repository layout

18_Temporal_KG_QA/
├── src/tkgqa/
│   ├── config.py  cli.py  logging_utils.py
│   ├── kg/          # schema (Fact / TemporalKG + verbalization), operators (parser + temporal ops)
│   ├── data/        # synth_tkg (generator), samples (seed), dataset, download
│   ├── retrieval/   # lexical (BM25 + RRF), encoder (dense), retriever (hybrid)
│   ├── models/      # model_registry
│   ├── training/    # train_retriever (MNRL), train_baseline, evaluate, tune, metrics
│   ├── agent/       # state, policy (D1-D5), tools, llm_orchestrator, tkgqa_agent
│   ├── api/         # schemas, dependencies, main (FastAPI), ui (Gradio), app_combined
│   ├── analysis/    # error_analysis, latency, per_type
│   ├── autoreport/  # artifact_loader, charts, report_pdf, slides_pptx
│   ├── monitoring/  # drift_report (job-log monitor)
│   ├── automation/  # autopilot (one button)
│   └── grading/     # checklist (rubric self-check)
├── configs/  docs/ (14 + DESIGN_BRIEF)  notebooks/  app/  deploy/  sample_data/  scripts/  tests/
├── Dockerfile  docker-compose.yml  Makefile  pyproject.toml  requirements*.txt  .github/workflows/ci.yml
└── LICENSE (MIT)  README.md

---

Models & data (all verified on the HF Hub)

Why a synthetic temporal-KG? The canonical temporal-KGQA benchmarks (CronQuestions, TimeQuestions) do not resolve on the HF Hub. The generator builds a KG of timed role facts with non-overlapping intervals (so point-in-time has a unique answer) + templated questions of every temporal type whose gold answers are computed by the same temporal operators the agent uses — giving free, exact retrieval supervision and isolating retrieval quality as the measured thing. The pure-python BM25 retriever stands in offline, so the agent, eval and tests run with no torch, no network.

---

Quickstart

pip install -e .                 # core: runs offline (BM25 retriever + temporal operators)
pip install -e .[all]            # + torch/sentence-transformers, FastAPI/Gradio, reportlab

tkgqa demo-agent --fast                                      # the 5-decision agent on seed questions
tkgqa ask --question "Who was the first CEO of Fairview?" --fast
tkgqa ask --question "Who was the president of Northland in 2005?" --fast
tkgqa evaluate --fast                                        # accuracy + per-type + Recall@k/MRR + abstention
tkgqa autopilot --no-train                                   # report.pdf + slides.pptx + grade + bundle
bash scripts/smoke.sh                                        # full offline smoke

--fast uses the BM25 retriever (no download). Drop it to use the fine-tuned bge-small dense retriever.

Train on Colab (H100, auto-adapts A100/L4/T4 — fine-tunes even on a free T4)

Push this folder to GitHub (or Drive), open notebooks/TKGQA_Colab_Training_H100_AUTOPILOT.ipynb, set the controls in cell 0, and Runtime → Run all. It fine-tunes the dense retriever (MNRL), runs the full evaluation, and writes report.pdf + slides.pptx + the bundle to Drive. See notebooks/COLAB_GUIDE.md.

The agent (the mandatory agentic component)

A deterministic FSM parse → retrieve → reason → select → abstain with five decision points:

The value-add over a pure similarity answer: explicit temporal reasoning + abstention + the supporting facts shown for provenance. An optional LLM brain (anthropic) is off by default.

Serving

tkgqa serve --ui             # FastAPI on :8000 + Gradio demo at /ui
# POST /ask  {question} -> answer + supporting timed facts + confidence + abstain flag
docker compose up --build    # containerized (text-only image)

Verified offline results

On a held-out synthetic temporal KG (offline, BM25 retriever + temporal operators): answer accuracy 0.93 vs the no-time-reasoning baseline 0.41 (explicit temporal reasoning more than doubles it); per-type before/after/first/last/when ~1.0, point-in-time 0.58 (limited by BM25 Recall@1 = 0.41, Recall@5 = 0.99) — exactly what the fine-tuned dense retriever lifts. Grade 1.0, all 5 decision points fire, 28 tests pass. On Colab the dense retriever raises Recall@1 and point-in-time accuracy.

Tests

pytest -q        # CPU-only, no downloads (HF_HUB_OFFLINE); the BM25 retriever stands in for the dense one

Documentation

Problem · Data · Data card · Models · Architecture · Agent · Evaluation · Deployment · Continual learning & monitoring · Privacy & robustness · Ethics · Project plan · Model card · Slides outline · Design brief

Ethics & license

Time-sensitive factual QA carries a stale-fact / temporal-misinformation risk: the tool assists, shows the supporting timed facts (provenance), and abstains when uncertain — it never asserts certainty on a "current" answer. KGs carry bias (ICEWS event-data skew, incompleteness); the supporting facts + confidence + decision trace make this auditable. Code is MIT (LICENSE); the shipped stack is permissive (bge-small MIT + a self-contained BM25); the non-commercial MultiTQ auxiliary is flagged and off by default.