📚 Citation Recommendation System

Recommend the papers a manuscript should cite. Give it a citation context (a sentence where a citation belongs) or a whole paper, and it returns a ranked list of papers to cite — via a two-stage prefetch → rerank pipeline (dense bi-encoder + BM25 + RRF → gated cross-encoder) wrapped in an agentic finite-state machine with local/global mode detection, self-citation filtering and an abstain gate.

NLP in Industry — Final Assignment. Author: Le Dinh Minh Quan (Student 23127460). Reference: nianlonggu/SciLit (ACL 2023) + Local Citation Recommendation w/ HAtten (Gu et al., ECIR 2022).

Keyword search misses paraphrased / conceptual citations and can't tell which paper a claim should cite. P09 fixes this: a fine-tuned context→cited-paper bi-encoder — supervised by real citation edges (the citation IS the label) — understands what kind of claim cites what kind of paper, and an agent turns a draft sentence into vetted, ranked citation suggestions (or honestly abstains).

---

✅ How this repo meets every assignment requirement

---

🏗️ Pipeline

citation context / query paper
  │  parse context · detect mode (LOCAL / GLOBAL)          ── D1 mode routing
  ▼
PREFETCH: dense (fine-tuned bi-encoder) ∥ BM25 → RRF fuse
  │  filter self-citations / already-cited / duplicates    ── D2 self-cite filter
  │  coverage gate → widen if the pool is thin             ── D3 coverage gate
  │  cross-encoder rerank (gated, top-K)                   ── D4 rerank gate
  ▼
confidence / abstain gate → "no confident citation"?       ── D5 abstain gate
  ▼
ranked papers to cite (+ reason, + optional citing sentence)

Two modes share one trained encoder: LOCAL (a citation-context sentence → the cited paper) and GLOBAL (a query paper → its reference set).

📦 Models & data (ids VERIFIED on the HF Hub)

The arXiv corpus has no citation edges — it is the candidate pool; the gold supervision comes from scirepeval citation triplets (+ synthetic LOCAL contexts). A bundled 46-paper seed corpus with a real citation graph powers the fully-offline demo, tests and eval.

🗂️ Repository layout

src/citerec/
├── config.py  cli.py  logging_utils.py
├── data/         samples.py (seed corpus + citation graph) · corpus.py · citation_pairs.py · download_dataset.py
├── models/       bm25.py · retriever.py · vector_store.py · reranker.py · model_registry.py
├── search/       hybrid.py (RRF) · engine.py (prefetch+rerank+filter) · expand.py
├── training/     train_retriever.py · train_reranker.py · evaluate.py · tune.py · metrics.py
├── agent/        state.py · policy.py (D1–D5) · tools.py · llm_orchestrator.py · citation_agent.py
├── api/          schemas.py · dependencies.py · main.py · ui.py · app_combined.py
├── analysis/ autoreport/ monitoring/ automation/ grading/
configs/ · data/ · models/ · tests/ · docs/ · notebooks/ · app/ · deploy/ · sample_data/

---

🚀 Quickstart

pip install -e ".[ml,api,report]"

citerec data                            # sanity-check corpus + citation + eval datasets
citerec demo-agent --tfidf              # run the agent on sample citation queries (offline)
citerec recommend --context "We retrieve passages with a dual-encoder dense retriever that beats BM25 [CITATION]."

Train

citerec --config configs/train.yaml train-retriever   # fine-tune the bi-encoder (MNRL, auto-resumes)
citerec --config configs/train.yaml train-reranker    # optional cross-encoder reranker (BCE)
citerec evaluate                                       # retriever vs BM25/zero-shot: Recall/MRR/MAP/nDCG

On Colab/GPU use the notebook (below) — it auto-profiles H100/A100/L4/T4 (batch = #in-batch negatives).

Serve

citerec serve --ui --port 7860          # FastAPI /recommend + /related + Gradio UI at /ui

One-button train → report + slides + self-grade

citerec autopilot --no-train            # eval → analysis → report.pdf + slides.pptx + bundle
citerec grade

---

🤖 The agent (mandatory agentic component)

A deterministic FSM with five decision points acting on the model's own intermediate outputs, plus an optional LLM brain (anthropic) that validates its output and falls back to rules:

• D1 mode routing (LOCAL sentence→paper vs GLOBAL paper→reference-set)
• D2 self-citation / already-cited / duplicate filter
• D3 prefetch-coverage gate (widen the search when the pool is thin)
• D4 rerank gate (run the cross-encoder only when the head is ambiguous)
• D5 confidence / abstain gate (return "no confident citation" rather than force a weak one)

Every step is timed + traced; same input + same index + brain disabled ⇒ identical output. See docs/agent_architecture.md.

☁️ Colab / H100 training

Open notebooks/Citation_Recommendation_Colab_Training_H100_AUTOPILOT.ipynb — mounts Drive, installs Colab-safe deps (never touches torch), auto-profiles the GPU (bf16/fp16 + CachedMNRL on T4), fine-tunes the retriever (+ reranker) resume-safely, evaluates vs BM25/zero-shot, runs the agent, and generates the report/slides. Step-by-step: notebooks/COLAB_GUIDE.md.

🧪 Tests

pytest -q        # CPU-only, no model/network downloads (seed corpus + BM25/TF-IDF + identity reranker)

📚 Docs index

docs/: problem_definition · data_description · data_card · model_selection · evaluation · agent_architecture · deployment · continual_learning_monitoring · privacy_robustness · project_plan · ethics_statement · architecture · model_card · slide_deck_outline · DESIGN_BRIEF.

📝 License

MIT — see LICENSE. Pretrained models keep their own licenses (table above); the scirepeval citation data is research-only (ODC-BY) — the trained model is for research/coursework. The arXiv corpus snapshot ends in 2021 — ingest newer papers incrementally for production.