LLM Fine-Tuning for Knowledge Graph Extraction & Clinical Summarization

---

Research project exploring LLM fine-tuning across two domains: (1) knowledge graph triple extraction from news articles using Mistral-7B + LoRA, and (2) clinical note summarization (Brief Hospital Course) using the MIMIC-IV-BHC dataset with LLaMA2, GPT-4, and Clinical-T5.

---

Table of Contents

• Overview
• Project Structure
• Experimental Results
• Technical Details
• Quick Start
• Dependencies
• Key Findings
• Future Work

---

Overview

This project investigates parameter-efficient fine-tuning (PEFT) techniques — specifically LoRA and QLoRA — applied to two distinct NLP tasks:

---

Project Structure

Fine_Tune_LLM/
├── Task1/
│   └── Task1.py                  # MedCAT entity extraction + Neo4j/RDF integration
├── Task2/
│   ├── newsfinetune1.py          # Case 1 — Gold Standard triples fine-tuning
│   ├── newfinetune2.py           # Case 2 — spaCy-filtered gold triples fine-tuning
│   └── newsfinetune3.py          # Case 3 — Combined gold + spaCy triples fine-tuning
├── data_preprocessing/
│   └── newKG21.py                # Preprocessing pipeline for NewsKG21 (all 3 cases)
├── mimic-iv-bhc.py               # Clinical BHC summarization pipeline
├── requirements.txt
└── TM2_Final_Report.pdf          # Full technical report

Task 1 — MedCAT & SNOMED CT Integration

Task1/Task1.py builds a medical entity extraction and knowledge graph storage pipeline:

• Loads a MedCAT model to identify SNOMED CT-coded entities in clinical text
• Exports entity relationships into Neo4j via the RDFLib framework
• Designed for downstream clinical knowledge graph construction

Task 2 — LLM Fine-Tuning for Knowledge Graph Extraction

Three experimental cases, each fine-tuning Mistral-7B-Instruct with LoRA (4-bit quantization):

All cases use an Alpaca-style instruction-input-response prompt format and are tracked via Weights & Biases.

Clinical Summarization Pipeline — MIMIC-IV-BHC

mimic-iv-bhc.py benchmarks Brief Hospital Course (BHC) summarization across:

• Adaptation strategies: Zero-shot prompting → Prefix prompting → QLoRA fine-tuning
• Models compared: Clinical-T5-Base, LLaMA2-13B-Chat, GPT-4 (with Flan-T5 as fallback)
• Evaluation: BERT Score (F1) across short, medium, and long clinical note bins
• Data split: Stratified 95% train / 5% test per context-length bin

Data Preprocessing

data_preprocessing/newKG21.py processes the raw NewsKG21 dataset into three prompt formats:

• Parses and cleans raw triple annotations
• Applies spaCy NLP for subject extraction and noise filtering
• Generates subject-conditioned prompts matched to each experimental case

---

Experimental Results

Knowledge Graph Triple Extraction (NewsKG21)

Fine-tuning Mistral-7B-Instruct on 1,501 training samples (80/20 split; 375 test samples) with evaluation on predicate-object pair extraction:

Best F1: Case 3 (Combined) — 0.2110
Best Precision: Case 2 (spaCy Filtered) — 0.2245

Clinical Summarization (MIMIC-IV-BHC) — BERT F1 Scores

Key finding: QLoRA fine-tuning yields substantial gains — +6.3% for Clinical-T5, +7.1% for LLaMA2.
LLaMA2-13B-Chat with QLoRA (0.683) outperforms the domain-specialized Clinical-T5 model.

---

Technical Details

Knowledge Graph Extraction Stack

Clinical Summarization Stack

---

Quick Start

Prerequisites

pip install -r requirements.txt
python -m spacy download en_core_web_sm

GPU required: CUDA-capable GPU with ≥16 GB VRAM recommended for Mistral-7B / LLaMA2.

---

Task 1 — MedCAT Entity Extraction

cd Task1
python Task1.py

Requires a local MedCAT model pack (SNOMED CT). Update model_path in the script before running.

---

Task 2 — Knowledge Graph Fine-Tuning

# Step 1: Preprocess the NewsKG21 dataset
cd data_preprocessing
python newKG21.py

# Step 2: Run fine-tuning experiments
cd ../Task2
python newsfinetune1.py   # Case 1 — Gold Standard
python newfinetune2.py    # Case 2 — spaCy Filtered
python newsfinetune3.py   # Case 3 — Combined

---

Clinical Summarization — MIMIC-IV-BHC

# Full pipeline (preprocessing + training + evaluation)
python mimic-iv-bhc.py --data-file mimic-iv-bhc.csv

# Inference only (skip training)
python mimic-iv-bhc.py --skip-training --context-bin all

# Specific context-length bin
python mimic-iv-bhc.py --context-bin short --output-dir results/short
python mimic-iv-bhc.py --context-bin medium --output-dir results/medium
python mimic-iv-bhc.py --context-bin long --output-dir results/long

Note: Access to the MIMIC-IV dataset requires a PhysioNet credentialed account.

---

Dependencies

# Core ML & LLM
pip install torch transformers datasets peft trl bitsandbytes
pip install "unsloth @ git+https://github.com/unslothai/unsloth.git"

# Clinical & NLP
pip install medcat spacy nltk bert-score openai

# Knowledge Graphs
pip install rdflib rdflib-neo4j neo4j

# Experiment tracking
pip install wandb huggingface-hub

# Utilities
pip install pandas numpy scikit-learn tqdm
python -m spacy download en_core_web_sm

Full pinned versions are in requirements.txt.

---

Key Findings

✅ Data filtering improves precision: Case 2 (spaCy Filtered) achieved the highest precision (0.2245), demonstrating that removing noisy triples yields more conservative but accurate extractions.

✅ Data augmentation improves F1: Case 3 (Combined) achieved the best overall F1 (0.2110), showing that augmenting gold triples with spaCy-extracted ones improves recall without sacrificing precision.

✅ QLoRA fine-tuning consistently outperforms prompting: Both Clinical-T5 and LLaMA2 gained 6–7% BERT F1 after QLoRA fine-tuning versus their zero-shot baselines.

✅ General-purpose LLMs can match domain-specific models: LLaMA2-13B-Chat with QLoRA (0.683) surpasses Clinical-T5-Base (0.647) despite not being trained on clinical text.

---

Future Work

• Enhanced evaluation: Fuzzy matching and semantic similarity metrics beyond strict string matching
• Data cleaning: Systematic correction of spelling errors and entity normalization in NewsKG21
• Stratified splits: Balanced train/test distributions across relation types
• Architecture exploration: Relation-specific BERT-based models for structured extraction
• Data augmentation: Synthetic training example generation via GPT-4
• Multi-task learning: Joint NER + relation extraction training
• Target performance: F1 improvement from ~0.21 to 0.35–0.40 for triple extraction

---

Dataset Information

---

Report

The full technical report is available in TM2_Final_Report.pdf.

---

Author: Mohan Adluru · June 2025