Code for paper: Large language models are powerful electronic health record encoders.
See the codeSerialize EHRs into plain text, embed them with general-purpose LLMs, and match a specialized EHR foundation model on clinical prediction without private medical pretraining data.
This repository contains the code for Large language models are powerful electronic health record encoders.
It is a fork of the EHRSHOT benchmark, and most of the pipeline, data handling, and baseline code is theirs. We kept the original code as unchanged as possible.
Figure 1. EHR records are serialized to text and embedded by a general-purpose LLM. A simple classifier is then trained per task and compared against the CLMBR-T-Base EHR foundation model and count-based baselines.
The core addition is a new feature-generation step, 4_generate_llm_features.py, that produces LLM embedding representations of the serialized EHR. This is the direct analogue of the existing count- and CLMBR-based features, so the rest of the EHRSHOT pipeline (few-shot sampling, evaluation, plotting) is reused unchanged. The main new components are:
llm_featurizer.py: the LLMFeaturizer that turns each patient timeline into a serialized record and aligns it with the benchmark labels.serialization/ehr_serializer.py and ehr_simple_serializer.py: the EHR-to-text serializers behind the different serialization strategies (list of recent codes, first occurrence, with or without time, and Markdown, JSON, XML, and YAML formats), selected via ehr_serializer_factory.py. Task instructions live in serialization/task_to_instructions_list.json.serialization/text_encoder.py: the TextEncoder and encoder classes that wrap each LLM embedding model (Qwen3-Embedding, GTE-Qwen2, LLM2Vec-Llama, the BERT-family baselines, and the newly added models below).We also added steps for the fine-tuning and in-context-learning experiments and for external validation on the UK Biobank (in UKB_validation/). Both are described in their own sections below.
All machine-specific paths and conda environments are read from a repo-root .env file. Copy the template and fill it in. The pipeline and launch scripts source it.
cp .env_template .env
# then set EHRSHOT_BENCHMARK_DIR, HUGGINGFACE_CACHE, the conda env names,
# and (for external validation) the UKB_* paths.
Set up the environment, install FEMR, and download the EHRSHOT dataset by following the upstream EHRSHOT instructions. Place the data under EHRSHOT_ASSETS/.
Install the LLM dependencies from requirements.txt (pinned for our main experiments with Qwen3-Emb). LLM2Vec-Llama3.1 needs a separate, incompatible dependency set, so use a dedicated env for it.
Reproduce the original EHRSHOT baselines with bash run_all.sh, as described upstream. We copied the data and results of this baseline run into EHRSHOT_ASSETS_BASELINES/ to keep them separate from the LLM experiments.
Run the LLM embedding experiments. Use a fresh EHRSHOT_ASSETS/ for these and add all EHRSHOT dataset files, then run the main entry point:
cd ehrshot/bash_scripts
bash run_experiments.sh # add --is_use_slurm to submit as SLURM jobs
The default configuration is Qwen3-Emb-8B with the unique_codes_list_recent_8k serialization. Uncomment entries in the text_encoders and serialization_strategies lists in run_experiments.sh to run other models (all Qwen3, Qwen2, and BERT variants, plus the two new model families below) and serializations. Each config runs 4_generate_llm_features.py and then the standard evaluation, mirroring the count and CLMBR pipeline.
Combine results and make figures with the helper scripts, for example 8_make_results_plots.py, 12_merge_and_plot_revision_results.py, and helper/create_plots.ipynb for the scaling and ablation figures.
Runtime note. Preprocessing (serializing all EHRs into text) is relatively slow and takes about 1 to 3 hours depending on hardware. Our setup regenerates the serialization for every run. If the same serialization is reused across experiments, caching it would save this cost.
During serialization, some medical codes are resolved to natural-language text using vocabulary files in ehrshot/custom_ontologies/. The CVX and ICD-10-PCS (PClassR) files are included in the repository. The CPT4 file is not redistributed here, so download it from the source below and save it as ehrshot/custom_ontologies/cpt4.csv. All sources are also listed in that folder's Readme.md.
ehrshot/custom_ontologies/cpt4.csv): https://gist.github.com/lieldulev/439793dc3c5a6613b661c33d71fdd185Beyond the frozen embedding plus classifier setup, we evaluate LoRA fine-tuning and decoder in-context learning (ICL). For any run, few-shot adaptation is either in-context examples or LoRA fine-tuning, never both. From ehrshot/bash_scripts/:
11_tune_finetuning_params.py (--plan-only, then sbatch --array, then --collect-only) over the grid in ehrshot/configs/tuning_grid.yaml, followed by select_best_params.py.10a_fit_and_eval_encoder.py (Qwen3-Emb-8B + LoRA) and 10b_fit_and_eval_decoder.py (Qwen3-8B + LoRA), run over (task, k, replicate).10b_fit_and_eval_decoder.py --icl_shots {0,2,4,6} (finetuning-free, use --batch_size 1 for long context).12_merge_and_plot_revision_results.py.The frozen winning configs are committed under ehrshot/configs/ (best_params_encoder.json, best_params_decoder.json). Neither fine-tuned variant surpassed the frozen Qwen3-Emb-8B baseline within the explored budgets.
Note. These scripts were refactored from the distributed pipeline we used for the actual runs (a Snakemake orchestration for scheduling and running the jobs) into plain scripts. The result is cleaner but was not executed in exactly this form, so new issues may have been introduced. Please open a GitHub Issue if you hit one and we are happy to help resolve it.
Code for the UK Biobank external validation is in UKB_validation/. It reuses the same list-based serialization (omitting units and values, which UKB lacks) and compares the LLM embedding model against CLMBR-T-Base, including the sensitivity analysis that restricts the LLM to CLMBR-mappable codes. Configure the UKB_* paths in .env assuming a local copy of the UK Biobank data.
After acceptance we added two further embedding-model families:
Both are integrated as standard encoders in serialization/text_encoder.py and run from run_experiments.sh (keys bioclinical_modernbert_{base,large} and harrier_oss_{270m,0_6b,27b}). The extended scaling plot below places them alongside the models from the paper. harrier-oss-v1-27b reaches the highest macro-AUROC of all tested models.
Extended scaling results including BioClinical ModernBERT and harrier-oss-v1 (see artifacts/figure_scaling_extended.png).
If you use this code, please cite our paper:
@article{Hegselmann2026llm,
title = {Large language models are powerful electronic health record encoders},
author = {Hegselmann, Stefan and von Arnim, Georg and Rheude, Tillmann and Kronenberg, Noel and Sontag, David and Hindricks, Gerhard and Eils, Roland and Wild, Benjamin},
journal = {npj Digital Medicine},
year = {2026},
volume = {9},
issn = {2398-6352},
doi = {10.1038/s41746-026-02915-9},
url = {https://doi.org/10.1038/s41746-026-02915-9}
}
Please also cite the original EHRSHOT benchmark this work builds on:
@article{wornow2023ehrshot,
title={EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models},
author={Michael Wornow and Rahul Thapa and Ethan Steinberg and Jason Fries and Nigam Shah},
year={2023},
eprint={2307.02028},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
The source code in this repository is released under the Apache License 2.0, following the upstream EHRSHOT repository. The EHRSHOT dataset and the CLMBR-T-Base model have their own licenses, listed on their respective pages.
Code for paper: Large language models are powerful electronic health record encoders.
See the codeSerialize EHRs into plain text, embed them with general-purpose LLMs, and match a specialized EHR foundation model on clinical prediction without private medical pretraining data.
This repository contains the code for Large language models are powerful electronic health record encoders.
It is a fork of the EHRSHOT benchmark, and most of the pipeline, data handling, and baseline code is theirs. We kept the original code as unchanged as possible.
Figure 1. EHR records are serialized to text and embedded by a general-purpose LLM. A simple classifier is then trained per task and compared against the CLMBR-T-Base EHR foundation model and count-based baselines.
The core addition is a new feature-generation step, 4_generate_llm_features.py, that produces LLM embedding representations of the serialized EHR. This is the direct analogue of the existing count- and CLMBR-based features, so the rest of the EHRSHOT pipeline (few-shot sampling, evaluation, plotting) is reused unchanged. The main new components are:
llm_featurizer.py: the LLMFeaturizer that turns each patient timeline into a serialized record and aligns it with the benchmark labels.serialization/ehr_serializer.py and ehr_simple_serializer.py: the EHR-to-text serializers behind the different serialization strategies (list of recent codes, first occurrence, with or without time, and Markdown, JSON, XML, and YAML formats), selected via ehr_serializer_factory.py. Task instructions live in serialization/task_to_instructions_list.json.serialization/text_encoder.py: the TextEncoder and encoder classes that wrap each LLM embedding model (Qwen3-Embedding, GTE-Qwen2, LLM2Vec-Llama, the BERT-family baselines, and the newly added models below).We also added steps for the fine-tuning and in-context-learning experiments and for external validation on the UK Biobank (in UKB_validation/). Both are described in their own sections below.
All machine-specific paths and conda environments are read from a repo-root .env file. Copy the template and fill it in. The pipeline and launch scripts source it.
cp .env_template .env
# then set EHRSHOT_BENCHMARK_DIR, HUGGINGFACE_CACHE, the conda env names,
# and (for external validation) the UKB_* paths.
Set up the environment, install FEMR, and download the EHRSHOT dataset by following the upstream EHRSHOT instructions. Place the data under EHRSHOT_ASSETS/.
Install the LLM dependencies from requirements.txt (pinned for our main experiments with Qwen3-Emb). LLM2Vec-Llama3.1 needs a separate, incompatible dependency set, so use a dedicated env for it.
Reproduce the original EHRSHOT baselines with bash run_all.sh, as described upstream. We copied the data and results of this baseline run into EHRSHOT_ASSETS_BASELINES/ to keep them separate from the LLM experiments.
Run the LLM embedding experiments. Use a fresh EHRSHOT_ASSETS/ for these and add all EHRSHOT dataset files, then run the main entry point:
cd ehrshot/bash_scripts
bash run_experiments.sh # add --is_use_slurm to submit as SLURM jobs
The default configuration is Qwen3-Emb-8B with the unique_codes_list_recent_8k serialization. Uncomment entries in the text_encoders and serialization_strategies lists in run_experiments.sh to run other models (all Qwen3, Qwen2, and BERT variants, plus the two new model families below) and serializations. Each config runs 4_generate_llm_features.py and then the standard evaluation, mirroring the count and CLMBR pipeline.
Combine results and make figures with the helper scripts, for example 8_make_results_plots.py, 12_merge_and_plot_revision_results.py, and helper/create_plots.ipynb for the scaling and ablation figures.
Runtime note. Preprocessing (serializing all EHRs into text) is relatively slow and takes about 1 to 3 hours depending on hardware. Our setup regenerates the serialization for every run. If the same serialization is reused across experiments, caching it would save this cost.
During serialization, some medical codes are resolved to natural-language text using vocabulary files in ehrshot/custom_ontologies/. The CVX and ICD-10-PCS (PClassR) files are included in the repository. The CPT4 file is not redistributed here, so download it from the source below and save it as ehrshot/custom_ontologies/cpt4.csv. All sources are also listed in that folder's Readme.md.
ehrshot/custom_ontologies/cpt4.csv): https://gist.github.com/lieldulev/439793dc3c5a6613b661c33d71fdd185Beyond the frozen embedding plus classifier setup, we evaluate LoRA fine-tuning and decoder in-context learning (ICL). For any run, few-shot adaptation is either in-context examples or LoRA fine-tuning, never both. From ehrshot/bash_scripts/:
11_tune_finetuning_params.py (--plan-only, then sbatch --array, then --collect-only) over the grid in ehrshot/configs/tuning_grid.yaml, followed by select_best_params.py.10a_fit_and_eval_encoder.py (Qwen3-Emb-8B + LoRA) and 10b_fit_and_eval_decoder.py (Qwen3-8B + LoRA), run over (task, k, replicate).10b_fit_and_eval_decoder.py --icl_shots {0,2,4,6} (finetuning-free, use --batch_size 1 for long context).12_merge_and_plot_revision_results.py.The frozen winning configs are committed under ehrshot/configs/ (best_params_encoder.json, best_params_decoder.json). Neither fine-tuned variant surpassed the frozen Qwen3-Emb-8B baseline within the explored budgets.
Note. These scripts were refactored from the distributed pipeline we used for the actual runs (a Snakemake orchestration for scheduling and running the jobs) into plain scripts. The result is cleaner but was not executed in exactly this form, so new issues may have been introduced. Please open a GitHub Issue if you hit one and we are happy to help resolve it.
Code for the UK Biobank external validation is in UKB_validation/. It reuses the same list-based serialization (omitting units and values, which UKB lacks) and compares the LLM embedding model against CLMBR-T-Base, including the sensitivity analysis that restricts the LLM to CLMBR-mappable codes. Configure the UKB_* paths in .env assuming a local copy of the UK Biobank data.
After acceptance we added two further embedding-model families:
Both are integrated as standard encoders in serialization/text_encoder.py and run from run_experiments.sh (keys bioclinical_modernbert_{base,large} and harrier_oss_{270m,0_6b,27b}). The extended scaling plot below places them alongside the models from the paper. harrier-oss-v1-27b reaches the highest macro-AUROC of all tested models.
Extended scaling results including BioClinical ModernBERT and harrier-oss-v1 (see artifacts/figure_scaling_extended.png).
If you use this code, please cite our paper:
@article{Hegselmann2026llm,
title = {Large language models are powerful electronic health record encoders},
author = {Hegselmann, Stefan and von Arnim, Georg and Rheude, Tillmann and Kronenberg, Noel and Sontag, David and Hindricks, Gerhard and Eils, Roland and Wild, Benjamin},
journal = {npj Digital Medicine},
year = {2026},
volume = {9},
issn = {2398-6352},
doi = {10.1038/s41746-026-02915-9},
url = {https://doi.org/10.1038/s41746-026-02915-9}
}
Please also cite the original EHRSHOT benchmark this work builds on:
@article{wornow2023ehrshot,
title={EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models},
author={Michael Wornow and Rahul Thapa and Ethan Steinberg and Jason Fries and Nigam Shah},
year={2023},
eprint={2307.02028},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
The source code in this repository is released under the Apache License 2.0, following the upstream EHRSHOT repository. The EHRSHOT dataset and the CLMBR-T-Base model have their own licenses, listed on their respective pages.