MyThesis

This repository contains the code, notebooks, and supporting utilities developed for a thesis project focused on the analysis of biomedical records. The workflow starts from the full dataset, isolates the natural-language notes, and evaluates multiple strategies for transforming those notes into information that can be analyzed more effectively.

Project Overview

The project is organized around four complementary stages:

1. Dataset exploration and preprocessing.
2. Automatic correction of the free-text notes.
3. Structured information extraction from the notes without rewriting them.
4. Topic modeling and exploratory analysis with BERTopic.

These stages are supported by shared utilities for model interaction, prompt construction, bootstrapping, path management, and evaluation.

Repository Structure

• 1_dataset_exploration/ contains notebooks used to inspect the dataset, identify the relevant subsets, and prepare the note-based text corpus.
• 2_correction/ contains the note-correction pipeline and its evaluation resources.
  • 0_100NotesSelection.ipynb supports the selection of notes used for correction experiments.
  • 1_annotate_notes_gui.py provides the annotation interface for manual review.
  • 2_correction_script.py runs the automated correction pipeline.
  • 3_evaluation/ contains the evaluation code and analysis notebooks for string-based and embedding-based comparison.
    • 1_string_scores/ contains metric computation for string-level evaluation.
    • 2_embedding/ contains embedding-based evaluation utilities and scripts.
    • Analysis/ contains notebooks used to study the impact of model family, model size, and prompting strategy.
• 3_extraction/ contains the structured extraction pipeline and its evaluation resources.
  • 1_extract_values_gui.py provides the interface for inspecting and extracting structured values.
  • 2_extraction_script.py runs the structured extraction pipeline.
  • 3_evaluation/ contains the evaluation code, normalization tools, and analysis notebooks for structured outputs.
    • Normalizations/ contains helpers for metric normalization and validation.
    • Analysis/ contains notebooks used to analyze model family, model size, and prompting effects.
• 4_bertopic/ contains notebooks for topic modeling and evaluation of the notes using BERTopic.
• utils/ provides shared functions, model wrappers, prompt templates, approaches, and bootstrapping utilities.
• images/ stores figures used in the project documentation and analysis.
• paths.py centralizes filesystem paths used throughout the project.

Main Workflow

The repository follows a progression from raw data inspection to model-assisted analysis. First, the dataset is explored to understand its attributes and the role of the natural-language notes. Next, two alternative LLM-based strategies are tested: one aims to correct the notes, while the other extracts structured variables directly from them. Both approaches are evaluated with dedicated metrics and analysis notebooks. Finally, BERTopic is applied to the notes to support unsupervised thematic exploration.

Reproducibility

The environment used to test the project is documented in requirements.txt, and the repository was tested with Python 3.9.6.

How to Use

From the repository root, create and activate a virtual environment, then install the dependencies:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

After the environment is ready, we suggest to choose the prompts and models you'd like to test: Correction prompts, Extraction prompts, Models.

Then, you can run the main pipeline scripts directly from the command line:

python 2_correction/2_correction_script.py
python 3_extraction/2_extraction_script.py
python utils/bootstrapping.py --task extraction
python utils/bootstrapping.py --task correction

The bootstrap command also accepts explicit paths when you want to override the defaults:

python utils/bootstrapping.py --scores-dir 3_extraction/extraction_files/llms_extraction_scores --output-path 3_extraction/extraction_files/bootstrapped_results.csv --n-bootstrap 10000

If you want to inspect or reproduce the exploratory analysis, you can re-run the notebooks in 1_dataset_exploration/ such as AccessDatasetsFiles.ipynb, dataset_attributes.ipynb, dataset_notes.ipynb, and dataset_signals.ipynb, together with the analysis notebooks in 2_correction/3_evaluation/Analysis/ and 3_extraction/3_evaluation/Analysis/ and the topic-modeling notebooks in 4_bertopic/.

The project paths are centralized in paths.py, so the scripts and notebooks can be run from the repository root without manually editing absolute paths.

Purpose

The goal of the project is to determine whether automated correction, structured extraction, and topic modeling can reveal clinically relevant patterns and improve the usability of semi-structured textual records for downstream analysis.