Medical Procedure Classifier

Professional Project — Multi-class classification of French medical procedure labels using TF-IDF and classical ML models.

What it does

Takes short French text descriptions of medical procedures (e.g. "Consultation cardiologue", "Extraction dentaire") and classifies them into 10 categories (Consultation, Dentaire, Hospitalisation, Optique, Pharmacie, …). The pipeline benchmarks several classifiers across character/word n-gram TF-IDF vectorizers, with experiments on oversampling, LLM-based data augmentation, unsupervised clustering audits to catch mislabelled rows, and targeted LLM-based rewriting/relabelling.

Reference result (unbiased, frozen clean holdout): LinearSVC + char n-grams (3,5) on dataset1.csv → F1 Macro 0.9609 / Accuracy 0.9497 — see reports/rapport_test_set_propre_dataset1.md and models/best_clean_holdout_model.joblib. The earlier 0.9618 figure below (LogisticRegression on the augmented+rewritten file) measured a test split partly composed of LLM-smoothed text and is kept for history only.

Quick start

git clone https://github.com/louisclzzzzzzz/Medical-Procedure-Classifier
cd medical-procedure-classifier
pip install -r requirements.txt

Python 3.10+ required. requirements.txt covers the core pipeline (TF-IDF + LinearSVC/LogisticRegression/RandomForest/MultinomialNB/XGBoost/LightGBM/CatBoost). Everything else (clustering audit, LLM rewriting, BERT/FastText experiments, cloud deploy) is optional and listed — commented out — at the bottom of the file; install only what the script you're running needs.

# Best configuration, single run
python main.py --name "LR Char 2-4" --model lr --vec char --ngram 2 4 --C 5.0

# Full grid search (~200 configs per model)
python scripts/tuning/tuning_all.py

FastText experiments only: download the French FastText model (cc.fr.300.bin) from fasttext.cc and place it in the project root (~7 GB, not included).

LOO-CV + LLM verification/rewriting only: requires a Mistral API key in the MISTRAL_API_KEY environment variable (or --api-key).

Project structure

.
├── main.py                       # CLI entry point — single experiment runs
├── requirements.txt
├── NEXT_STEPS.md                 # Open work items / roadmap (French)
├── data/
│   ├── dataset1.csv / dataset2.csv                  # Base labelled datasets
│   ├── dataset{1,2}_hospmat_fixed.csv                # Hospitalisation/Maternité rule fixes applied
│   ├── dataset{1,2}_final.csv                        # Clustering + LOO-CV + targeted LLM cleaned — reference datasets
│   ├── dataset{1,2}_gen.csv / dataset{1,2}_augmented.csv  # Synthetic augmentation experiments (superseded)
│   ├── dataset{1,2}_rewritten.csv                    # LOO-CV+LLM rewriting, pre-clustering-pipeline (superseded)
│   ├── dataset1_augmented_rewritten.csv              # Augmented + LLM-rewritten ambiguous labels (dataset1 only)
│   └── raw/                                          # Original Excel exports the CSVs were built from
├── models/                       # Trained models (.joblib, gitignored — see "Models" below)
├── notebooks/
│   └── camembert_bio_colab.ipynb # Colab notebook: CamemBERT-bio fine-tuning (needs GPU) — exploratory
├── scripts/
│   ├── analysis/    # clustering_pipeline.py, loo_cv_mistral.py, rewrite_flagged.py,
│   │                # eval_clean_holdout.py, grid_search_clean_holdout.py, plot_learning_curve.py, _best_report.py
│   ├── tuning/      # tuning_all.py, fasttext_tuning.py, bert_tuning.py, camembert_bio_tuning.py,
│   │                # smote_tuning.py, adasyn_tuning.py, run_combined_tuning.py, run_top10_combined.py
│   ├── data/        # build_dataset.py, extract_data.py — historical raw→CSV extraction (superseded)
│   └── deploy/      # launch_sagemaker.py, launch_vertexai.py — cloud training launchers for camembert_bio_tuning.py
├── results/                      # Run logs, metric summaries, clustering/LOO-CV outputs (CSV, PNG)
│   ├── synthèse_runs.csv         # Default grid-search log for `scripts/tuning/tuning_all.py` (no --csv override)
│   ├── manual_runs.csv           # Log of ad hoc `python main.py` single-experiment runs
│   ├── syntheses/                # synthèse_<experiment>_runs.csv — grid search summaries for named experiments (--csv override)
│   ├── clustering*/              # HDBSCAN+UMAP audit outputs per dataset/parameter setting
│   ├── loo_cv/                   # LOO-CV results, flagged errors, LLM verification logs/cache
│   └── learning_curves/
├── reports/                      # Full write-ups (French)
│   ├── rapport.md                            # Baseline, augmentation, SMOTE/ADASYN, LLM rewriting
│   ├── rapport_clustering.md                 # Unsupervised clustering audit methodology
│   ├── rapport_nettoyage_dataset1.md         # dataset1 cleaning pipeline write-up
│   ├── rapport_nettoyage_dataset2.md         # dataset2 cleaning pipeline write-up
│   └── rapport_test_set_propre_dataset1.md   # Unbiased clean holdout evaluation methodology
├── archive/                       # Superseded/legacy material, kept for provenance — not in git (see archive/README.md)
├── cc.fr.300.bin                  # FastText embeddings — not included, download separately (see Quick start)
└── venv/                          # Local virtualenv — not included

Data

• dataset1 — 1 166 labelled examples (10 classes), strongly imbalanced (Autres 37 % vs Maternité 1.5 %)
• dataset2 — 4 697 labelled examples (10 classes), same label set, larger and noisier
• data/dataset1.csv / data/dataset2.csv — base datasets
• data/dataset{1,2}_hospmat_fixed.csv — Hospitalisation/Maternité boundary disambiguated by rule (see cleaning reports below)
• data/dataset{1,2}_final.csv — cleaned datasets: clustering audit (HDBSCAN+UMAP) ∩ LOO-CV errors → targeted Mistral Small verification → rewrite/relabel. Current reference datasets for tuning.
• data/dataset{1,2}_gen.csv / data/dataset{1,2}_augmented.csv / data/dataset1_augmented_rewritten.csv / data/dataset{1,2}_rewritten.csv — earlier synthetic-augmentation and LLM-rewriting experiments, superseded by the clean holdout / *_final findings above
• data/raw/ — the original Excel exports (DataSet annoté.xlsx, COURTAGE - TRANSCO LIBELLE ACT SANTE - {2022,2023,2024,2025}.xlsx, …) that dataset1.csv/dataset2.csv were built from. Kept for provenance; the CSVs in data/ are the source of truth for all scripts.

Pipeline overview

Raw labels
  └─ TF-IDF (char n-grams)        →  LinearSVC / LR / RF / NB  →  grid search
  └─ SMOTE / ADASYN               →  oversampling experiments (§6, abandoned)
  └─ LLM generation                →  synthetic samples for rare classes (§7)
  └─ LOO-CV + Mistral Small        →  flag ambiguous labels → rewrite → retune (§8)
  └─ Clustering (HDBSCAN+UMAP)     →  geometric outlier/mismatch audit
       ∩ LOO-CV errors             →  double-signal priority list → targeted LLM verification
       → dataset{1,2}_final.csv   →  re-tune → unbiased frozen clean holdout evaluation

Results summary

SMOTE and ADASYN both degraded performance (down to −0.11 F1 Macro) — see §6 of reports/rapport.md.

Baseline top 4 (reports/rapport.md §4–5)

Unbiased clean holdout grid search (474 runs, dataset1)

Full details: reports/rapport_test_set_propre_dataset1.md.

Cleaning pipeline (clustering + LOO-CV + targeted LLM)

A second cleaning pipeline catches annotation errors geometrically, instead of relying solely on synthetic augmentation:

1. Unsupervised clustering audit — sentence-transformer embeddings + UMAP + HDBSCAN flag rows whose label disagrees with the geometric structure of the embedding space (outliers + cluster-label mismatches). See reports/rapport_clustering.md.
2. Hospitalisation/Maternité rule-based disambiguation — these two classes account for most of the semantic boundary overlap (lexical rules on obstetric vocabulary, "chambre particulière", anesthesia context, etc.).
3. LOO-CV flags rows the model itself misclassifies when held out.
4. Double-signal prioritisation — only rows flagged by both clustering and LOO-CV go through (costly) LLM verification with Mistral Small, cutting API calls by ~85% via cache reuse.
5. Dataset reconstruction — confirmed-ambiguous rows are rewritten, model-confirmed mislabels are relabelled, LLM-proposed third categories are applied on high confidence.

Full write-ups: reports/rapport_nettoyage_dataset1.md (1 166 rows, +7.2 F1 macro vs raw baseline, no augmentation needed) and reports/rapport_nettoyage_dataset2.md (4 697 rows, +10.4 F1 macro vs raw).

Models (models/)

Not tracked in git (*.joblib is gitignored) — regenerate by re-running the script that produced it, or use the copies shipped in the handoff package directly. Each experiment writes its own file, so the folder doubles as a history of the project. The three that matter for actually using the classifier:

The rest are byproducts of earlier or abandoned experiments, kept for comparison: best_model.joblib, best_rf_model.joblib (early baseline checkpoints), best_smote_model.joblib / best_adasyn_model.joblib (oversampling, abandoned — see §6), best_augmented_model.joblib / best_augmented_model_v2.joblib / best_rewritten_model.joblib / best_model_dataset2_rewritten.joblib (synthetic augmentation / LLM-rewriting, pre-clustering-pipeline), best_model_dataset2.joblib (dataset2 before cleaning), best_fasttext_model.joblib (exploratory, see below), last_run_model.joblib (whatever python main.py produced last). best_combined_model.joblib belongs to the abandoned "combined dataset" experiment (see below) — it is large (~216 MB) and not required for anything documented here.

Usage

Note: several scripts (tuning_all.py, fasttext_tuning.py, loo_cv_mistral.py, plot_learning_curve.py) default their --model argument to a bare filename (e.g. best_model.joblib) written to/read from the project root, not models/. This is a long-standing quirk, not a documented convention — pass an explicit --model models/... path (as in the examples below) rather than relying on the default.

Single experiment

python main.py --name "LR Char 2-4" --model lr --vec char --ngram 2 4 --C 5.0
python main.py --name "SVC Char 3-5" --model svc --vec char --ngram 3 5 --C 1.0

Grid search

python scripts/tuning/tuning_all.py
# With a custom dataset and output paths:
python scripts/tuning/tuning_all.py --dataset data/dataset1_final.csv \
                              --csv results/syntheses/synthèse_dataset1_final_runs.csv \
                              --model models/best_model_dataset1_final.joblib

LOO-CV + LLM rewriting (§8 pipeline)

python scripts/analysis/loo_cv_mistral.py     # LOO-CV, flag errors, rewrite with Mistral Small
python scripts/analysis/rewrite_flagged.py    # Apply rewrites to produce dataset1_augmented_rewritten.csv
python scripts/tuning/tuning_all.py --dataset data/dataset1_augmented_rewritten.csv

Clustering audit + clean dataset construction

python scripts/analysis/clustering_pipeline.py --min-cluster-size 15 --output-dir results/clustering_d1_mcs15
python scripts/analysis/loo_cv_mistral.py --dataset data/dataset1_hospmat_fixed.csv \
                                           --model models/best_model.joblib --no-llm

Unbiased clean holdout evaluation

python scripts/analysis/eval_clean_holdout.py        # Single config, frozen clean test set
python scripts/analysis/grid_search_clean_holdout.py # Full grid search on the same frozen split

Exploratory tracks (not yet conclusive)

These are legitimate open avenues, kept and documented rather than deleted — none has yet beaten the TF-IDF + LinearSVC baseline, so none is the recommended path today.

• Embeddings vs TF-IDF — scripts/tuning/fasttext_tuning.py (FastText, needs cc.fr.300.bin), scripts/tuning/bert_tuning.py and scripts/tuning/camembert_bio_tuning.py (needs torch/transformers, GPU recommended), and notebooks/camembert_bio_colab.ipynb (same experiment, runnable on Colab's free GPU). See item 8 of NEXT_STEPS.md.
• Cloud training launchers — scripts/deploy/launch_sagemaker.py and scripts/deploy/launch_vertexai.py submit camembert_bio_tuning.py as a remote GPU job (AWS SageMaker / GCP Vertex AI respectively). Require their own credentials/setup — see each script's docstring.
• "Combined dataset" experiment — scripts/tuning/run_combined_tuning.py / run_top10_combined.py and models/best_combined_model.joblib targeted a merged dataset1+dataset2+TRANSCO-2025 dataset built by scripts/data/build_dataset.py. Abandoned: the intermediate file it depends on (data/dataset_combined.csv) was never carried forward, and the approach was superseded by cleaning dataset1/dataset2 independently. Kept for reference only.

Reports (reports/)

All write-ups are in French:

• rapport.md — full experimental report: baseline, SMOTE/ADASYN, synthetic augmentation, LOO-CV + LLM rewriting
• rapport_clustering.md — unsupervised clustering audit methodology
• rapport_nettoyage_dataset1.md — dataset1 cleaning pipeline write-up
• rapport_nettoyage_dataset2.md — dataset2 cleaning pipeline write-up
• rapport_test_set_propre_dataset1.md — unbiased clean holdout evaluation methodology

Archive (archive/)

Superseded material kept on disk for provenance but not required to reproduce anything above, and not tracked in git: the original MATLAB prototype, an old ad hoc backup zip, and one unreferenced/messy raw dataset. Details and rationale for each item: archive/README.md.

Next steps

Open work items and rationale for what to try next: NEXT_STEPS.md.

License

MIT