Hierarchy-Aware Multimodal Unlearning for Medical AI

Authors

Fengli Wu, Vaidehi Patil, Jaehong Yoon, Yue Zhang, Mohit Bansal

Overview

Pretrained Multimodal Large Language Models (MLLMs) are increasingly used in sensitive domains such as medical AI, where privacy regulations like HIPAA and GDPR require specific removal of individuals' or institutions' data. This motivates machine unlearning, which aims to remove the influence of target data from a trained model. However, existing unlearning benchmarks fail to reflect the hierarchical and multimodal structure of real-world medical data, limiting their ability to properly evaluate unlearning in practice. Therefore, we introduce MedForget, a hierarchy-aware multimodal unlearning benchmark that models hospital data as a nested structure, enabling fine-grained evaluation of multimodal unlearning across retain and forget splits.

Experiments with current unlearning methods show that existing approaches struggle to achieve effective hierarchy-aware forgetting without degrading downstream medical utility. To address this limitation, we propose Cross-modal Hierarchy-Informed Projection for unlearning (CHIP), a training-free, hierarchy-aware multimodal unlearning method that deletes information by selectively removing target-specific weight subspaces while preserving sibling-shared information. Experiments show that CHIP achieves the highest forget-retain performance gap across all hierarchy levels while maintaining competitive downstream utility compared to existing methods.

Requirements

pip install torch transformers rouge-score openai peft pandas pillow python-dotenv qwen-vl-utils

Set LLM API key (for factuality evaluation):

export DEEPSEEK_API_KEY="your-api-key-here"

Data

The MedForget benchmark is built on MIMIC-CXR. The dataset and fine-tuned model checkpoint will be released on PhysioNet soon. Stay tuned!

Fine-tuning

We fine-tune Lingshu-7B (a Qwen2.5-VL based medical MLLM) on the MedForget training set using LoRA.

Quick Start

python finetune.py \
    --model_id lingshu-medical-mllm/Lingshu-7B \
    --data_dir /path/to/training_data.parquet \
    --save_dir /path/to/save_dir \
    --batch_size 4 \
    --num_epochs 10 \
    --lr 1e-4 \
    --gradient_accumulation_steps 4

Training automatically resumes from the latest checkpoint if interrupted.

Fine-tuning Hyperparameters

CHIP: Our Method

CHIP is a training-free unlearning method. It requires only forward passes to collect activations, then modifies model weights via orthogonal projection — no gradient updates needed.

Quick Start

./run_chip.sh --model /path/to/finetuned_model --data /path/to/medforget_data --level institution_level

To run at other hierarchy levels:

./run_chip.sh -m /path/to/model -d /path/to/data -l patient_level
./run_chip.sh -m /path/to/model -d /path/to/data -l study_level
./run_chip.sh -m /path/to/model -d /path/to/data -l section_level

Or call the Python entry point directly:

python chip/chip.py \
    --model_id lingshu-medical-mllm/Lingshu-7B \
    --vanilla_dir /path/to/finetuned_model \
    --model_type Lingshu \
    --forget_file data/institution_level/forget_set_all.parquet \
    --retain_file data/institution_level/retain_set_all.parquet \
    --target_level institution \
    --save_dir ./chip_output

CHIP Hyperparameters

Evaluation

Basic Usage

# Evaluate both forget and retain sets
./run_eval.sh --model /path/to/model --level patient_level --dataset both

With LoRA Adapter

./run_eval.sh \
  --model /path/to/adapter \
  --base-model /path/to/base/model \
  --level study_level \
  --batch-size 32

Direct Python Script

python eval.py \
  --model_path /path/to/model \
  --data_path /path/to/data.parquet \
  --output_dir results \
  --inference_batch_size 8

Evaluation Arguments

Output

Results are saved in eval_results_LEVEL_TIMESTAMP/:

• detailed_results.json: Per-sample scores (ROUGE-L, factuality, total)
• evaluation_summary.json: Statistical summary with mean, std, and percentiles

Citation

@article{wu2025medforget,
    title={Hierarchy-Aware Multimodal Unlearning for Medical AI},
    author={Fengli Wu and Vaidehi Patil and Jaehong Yoon and Yue Zhang and Mohit Bansal},
    journal={arXiv preprint arXiv:2512.09867},
    year={2025},
    url={https://arxiv.org/abs/2512.09867}
}