picolimpid/HyDE-Fine-Tuning-for-Efficient-Query-Retrieval

stars

commits

Python

primary language

Apr 12, 2026

updated

README

HyDE Fine-Tuning for Efficient Query Retrieval

Portfolio code for HyDE-style dense retrieval on 2WikiMultihopQA and a one-shot query encoder trained to match mean HyDE teacher embeddings (no LLM at retrieval time).

CreatLab · Aug–Nov 2025 · Champaign, IL

Narrative (matches resume)

HyDE RAG — Hypothetical documents from an LLM are embedded and mean-pooled with the query; retrieval is dense similarity (FAISS) over a 2Wiki corpus.
Distilled “one-shot HyDE” — Fine-tune the query tower (E5-Mistral + LoRA, or Contriever) so its query embedding predicts the teacher vector = mean embedding of those hypotheses. The retrieval stack (index, metrics) stays the same; only the query side changes at inference.
What we saw — Small / limited gains on instruction-tuned bi-encoders (E5-Mistral-Instruct); larger Recall@K gains on symmetric encoders (Contriever, BGE-style). Reproduce with the eval scripts below on your split.

Repository map

Stage	What to run	Location
A. Corpus / queries	BEIR-style `corpus.jsonl`, `queries.jsonl`, `qrels.tsv`	`two_wiki/data/data/dev_retrieval/` (create locally; see `two_wiki/prepare_2wiki_corpus.py`)
B. Hypotheses (LLM)	Export `hypothesis_docs` per query	`two_wiki/export_hyde_hypotheses.py` + `two_wiki/hyde_simple_config.json`
C. Teacher vectors	Mean-pool hypothesis embeddings → `.npy` + `.json`	`python scripts/build_teacher_embeddings.py` (config: `config/e5_embedding_config.json`)
D. Distillation	E5 query encoder (LoRA) or Contriever student	`training/distill_e5_query_encoder.py`, `training/distill_contriever.py`
E. Merge LoRA (E5)	Full weights for indexing	`training/merge_lora.py`
F. Indexes & metrics	FAISS build / Recall@K	`two_wiki/build__index.py`, `two_wiki/eval_.py`, `two_wiki/bge/`
G. End-to-end HyDE demos	Live LLM + encoder + FAISS on dev split	`retrieval/hyde_rag_bge_m3_2wiki.py`, `retrieval/hyde_rag_e5_2wiki.py`
H. Extra baselines	Contriever TreeHop-style eval	`retrieval/eval_contriever_2wiki.py`, `retrieval/simple_2wiki_multihop_eval.py`, `retrieval/build_2wiki_index.py`

Core library: src/hyde/ (Promptor, generators, HyDE mean-pooling).

Setup

cd "HyDE Fine-Tuning for Efficient Query Retrieval"
python -m venv .venv && source .venv/bin/activate
pip install -e .
pip install -r requirements.txt

Run scripts from the repository root unless noted (two_wiki/* is usually run from two_wiki/ so relative data/ paths resolve).

Environment variables (optional)

Variable	Used by
`HYDE_E5_LORA_PATH`	`training/merge_lora.py`, `two_wiki/build_e5_index.py`, `two_wiki/eval_e5_retrieval.py`
`HYDE_MERGED_E5_DIR`	`training/merge_lora.py` output
`HYDE_2WIKI_DATA_DIR`	`two_wiki/build_e5_index.py`, `two_wiki/eval_e5_retrieval.py`
`HYDE_E5_INDEX_FILE`	`two_wiki/eval_e5_retrieval.py`

Teacher embeddings script

# After editing config/e5_embedding_config.json → metadata_file points to your JSONL/JSON export
python scripts/build_teacher_embeddings.py --config config/e5_embedding_config.json
# Optional: --backend transformers_e5  (HF E5 + last-token pool instead of SentenceTransformer)

Outputs under artifacts/teacher_embeddings/, including stable names teacher_embeddings.npy and teacher_metadata.json for training defaults.

What was removed (on purpose)

MS MARCO / DL19 demos, duplicate embedding generators, scratch notebooks, Qwen index, duplicate E5 eval copies, and one-off test.py files—so the tree tracks the 2Wiki + distillation story only.

中文说明

目录已按 简历主线 重组：假文档生成 → 教师向量 → 查询端蒸馏 → 建索引与 Recall 评估；重复脚本已删除，两个旧的 hyde_embedding_generator*.py 合并为 scripts/build_teacher_embeddings.py。

License

Add a LICENSE when you open-source; upstream models (E5, BGE, Contriever) follow their own terms.

Contributors

picolimpid

1 commits

Similar repos

yuancu/zc_disti

Adefioye/vector-search-experiment

manveertamber/cadet-dense-retrieval

manveertamber/enhancing_domain_adaptation

Alrope123/compactds-retrieval

ielab/diffretriever

IBM/task-aware-embedding-refinement

Test-time optimization of query embedding representations for zero-shot classification and retrieval

NovaSearch-Team/RAG-Retrieval

Unify Efficient Fine-tuning of RAG Retrieval, including Embedding, ColBERT, ReRanker.

1,129

Languages

Python

97.3%

Shell

2.7%

picolimpid/HyDE-Fine-Tuning-for-Efficient-Query-Retrieval

stars

commits

Python

primary language

Apr 12, 2026

updated

README

HyDE Fine-Tuning for Efficient Query Retrieval

Portfolio code for HyDE-style dense retrieval on 2WikiMultihopQA and a one-shot query encoder trained to match mean HyDE teacher embeddings (no LLM at retrieval time).

CreatLab · Aug–Nov 2025 · Champaign, IL

Narrative (matches resume)

HyDE RAG — Hypothetical documents from an LLM are embedded and mean-pooled with the query; retrieval is dense similarity (FAISS) over a 2Wiki corpus.
Distilled “one-shot HyDE” — Fine-tune the query tower (E5-Mistral + LoRA, or Contriever) so its query embedding predicts the teacher vector = mean embedding of those hypotheses. The retrieval stack (index, metrics) stays the same; only the query side changes at inference.
What we saw — Small / limited gains on instruction-tuned bi-encoders (E5-Mistral-Instruct); larger Recall@K gains on symmetric encoders (Contriever, BGE-style). Reproduce with the eval scripts below on your split.

Repository map

Stage	What to run	Location
A. Corpus / queries	BEIR-style `corpus.jsonl`, `queries.jsonl`, `qrels.tsv`	`two_wiki/data/data/dev_retrieval/` (create locally; see `two_wiki/prepare_2wiki_corpus.py`)
B. Hypotheses (LLM)	Export `hypothesis_docs` per query	`two_wiki/export_hyde_hypotheses.py` + `two_wiki/hyde_simple_config.json`
C. Teacher vectors	Mean-pool hypothesis embeddings → `.npy` + `.json`	`python scripts/build_teacher_embeddings.py` (config: `config/e5_embedding_config.json`)
D. Distillation	E5 query encoder (LoRA) or Contriever student	`training/distill_e5_query_encoder.py`, `training/distill_contriever.py`
E. Merge LoRA (E5)	Full weights for indexing	`training/merge_lora.py`
F. Indexes & metrics	FAISS build / Recall@K	`two_wiki/build__index.py`, `two_wiki/eval_.py`, `two_wiki/bge/`
G. End-to-end HyDE demos	Live LLM + encoder + FAISS on dev split	`retrieval/hyde_rag_bge_m3_2wiki.py`, `retrieval/hyde_rag_e5_2wiki.py`
H. Extra baselines	Contriever TreeHop-style eval	`retrieval/eval_contriever_2wiki.py`, `retrieval/simple_2wiki_multihop_eval.py`, `retrieval/build_2wiki_index.py`

Core library: src/hyde/ (Promptor, generators, HyDE mean-pooling).

Setup

cd "HyDE Fine-Tuning for Efficient Query Retrieval"
python -m venv .venv && source .venv/bin/activate
pip install -e .
pip install -r requirements.txt

Run scripts from the repository root unless noted (two_wiki/* is usually run from two_wiki/ so relative data/ paths resolve).

Environment variables (optional)

Variable	Used by
`HYDE_E5_LORA_PATH`	`training/merge_lora.py`, `two_wiki/build_e5_index.py`, `two_wiki/eval_e5_retrieval.py`
`HYDE_MERGED_E5_DIR`	`training/merge_lora.py` output
`HYDE_2WIKI_DATA_DIR`	`two_wiki/build_e5_index.py`, `two_wiki/eval_e5_retrieval.py`
`HYDE_E5_INDEX_FILE`	`two_wiki/eval_e5_retrieval.py`

Teacher embeddings script

# After editing config/e5_embedding_config.json → metadata_file points to your JSONL/JSON export
python scripts/build_teacher_embeddings.py --config config/e5_embedding_config.json
# Optional: --backend transformers_e5  (HF E5 + last-token pool instead of SentenceTransformer)

Outputs under artifacts/teacher_embeddings/, including stable names teacher_embeddings.npy and teacher_metadata.json for training defaults.

What was removed (on purpose)

中文说明

License

Add a LICENSE when you open-source; upstream models (E5, BGE, Contriever) follow their own terms.

Similar repos

yuancu/zc_disti

Adefioye/vector-search-experiment

manveertamber/cadet-dense-retrieval

manveertamber/enhancing_domain_adaptation

Alrope123/compactds-retrieval

ielab/diffretriever

IBM/task-aware-embedding-refinement

Test-time optimization of query embedding representations for zero-shot classification and retrieval

NovaSearch-Team/RAG-Retrieval

Unify Efficient Fine-tuning of RAG Retrieval, including Embedding, ColBERT, ReRanker.

1,129

Contributors

picolimpid

1 commits

Languages

Python

97.3%

Shell

2.7%