Diagnosis Matching Project

Matching pairs of free-text Russian medical diagnoses using a meta-model built on NLP heuristics, LLM prompting, RAG, and embedding similarity.

Project Structure

diagnosis-matching-project/
├── data/       # All unique datasets (canonical source of truth)
├── core/       # Production pipeline — feature engineering + AutoGluon meta-model
├── ablation/   # Ablation study — evaluates contribution of each feature group
└── distil/     # Knowledge distillation — compresses meta-model into a single transformer

Dependency Flow

core ──(feature CSVs)──▶ ablation
core ──(teacher probs)──▶ distil

Links are data-only — no code is shared between components. Run core first to produce the datasets that ablation and distil consume.

Note: All data paths in component scripts should be set relative to the root data/ folder, not the component's own data/ subdirectory.

Setup

Each component is an independent uv project. Install all three:

cd core     && uv sync    # Python 3.11, 143 packages, torch+cu126
cd ablation && uv sync    # Python 3.10+, 34 packages
cd distil   && uv sync    # Python 3.13, 85 packages, torch+cu126

Run any script inside a component:

cd core && uv run python pipeline.py

Data (data/)

All unique datasets live in the root data/ folder. Files are prefixed by component.

Datasets are stored on HuggingFace. Download before running any component:

huggingface-cli download sm1rk/diagnosis-matching-project-data --repo-type dataset --local-dir data/

Or via Python:

from huggingface_hub import snapshot_download
snapshot_download("sm1rk/diagnosis-matching-project-data", repo_type="dataset", local_dir="data/")

data/
├── final-test.csv                              # 55 MB  — core: raw test pairs (272K rows)
├── final-test-preprocessed.csv                 # 77 MB  — core: cleaned test pairs
├── rag-diagnosis2icd.json                      # 2.2 MB — core: diagnosis → ICD-10 mapping (~16K entries)
├── rag-bergamot-index.faiss                    # 51 MB  — core: FAISS embedding index for RAG
├── rag-bergamot-index.pkl                      # 3.5 MB — core: FAISS index metadata
├── distil-gold.csv                             # 538 KB — distil: human-curated labels (4.8K rows)
├── distil-silver.csv                           # 33 MB  — distil: teacher-labeled pairs (272K rows)
├── distil-test.csv                             # 305 KB — distil: test with features (1.5K rows)
├── distil-train-full.csv                       # 33 MB  — distil: gold + silver combined
├── distil-train-isgold-full.csv                # 33 MB  — distil: full train + is_gold flag
├── distil-train-nondup.csv                     # 7.8 MB — distil: deduped train (57K rows)
├── distil-train-isgold-nondup.csv              # 7.7 MB — distil: deduped + is_gold
├── distil-train-isgold-preprocessed-nondup.csv # 7.5 MB — distil: final training data
├── ablation-train.csv                          # 567 KB — ablation: feature-engineered train
└── ablation-test.csv                           # 171 KB — ablation: feature-engineered test

Components

core/ — Meta-Pipeline

Trains an AutoGluon LightGBM meta-model on 16 engineered features spanning four sectors:

ablation/ — Ablation Study

Trains separate AutoGluon models on feature subsets to measure each sector's contribution. Experiments are defined in run_ablation.py and run individually:

cd ablation && uv run python run_ablation.py full
cd ablation && uv run python run_ablation.py no_nlp

distil/ — Knowledge Distillation

Distills the 16-feature meta-model into a single ModernBERT transformer for binary sequence classification, using teacher probabilities as soft labels.

cd distil && uv run python scripts/distil.py            # training particular model
cd distil && uv run python scripts/distil-eval.py       # evaluation
cd distil && uv run python scripts/distil-hpo.py        # hyperparameter search
cd distil && uv run python scripts/distil-inference.py  # inference