Expert-Guided Prompting and Retrieval-Augmented Generation for EMS Question Answering

Code repository: Expert-Guided Prompting and Retrieval-Augmented Generation for EMS Question Answering, AAAI 2026

🌐 Project · 📄 Paper · 🤗 Dataset

Overall Stucture of the Work

1. We introduce EMSQA, the first EMS MCQA dataset of 24.3K questions, curated based on public and private sources, covering 10 subject areas and 4 certification levels, and accompanied by a structured, subject area-aligned EMS knowledge base (KB) with 40K documents and 4M real-world patient care reports. Partial data (from public sources) and the whole EMS KB will be shared as a resource with the EMS and research communities.
2. We propose two approaches to inject domain expertise into LLMs: 1) an expertise-guided prompting strategy (Expert-CoT) that encourages step-by-step reasoning from a domain-specific perspective. 2) an expertise-guided RAG method (ExpertRAG) that selectively retrieves expertise-aligned knowledge from curated EMS KBs and patient records
3. We benchmark multiple LLMs on EMSQA, evaluating performance across certification levels and subject areas, and compare our framework against SOTA RAG methods. Experimental results show that combining Expert-CoT and ExpertRAG yields up to a 4.67% improvement in accuracy. Notably, the 32B expertise-augmented models pass all the EMS certification simulation exams.

Data, Knowledge Base and Patient Records

• Dataset

  Download from Huggingface EMS-MCQA

• Knowledge base

  Download from Huggingface EMS-Knowledge. We also upload the embedding encoded by MedCPT to link, where you can download and directly load using FAISS.

• Patient Records

  Apply at NEMSIS to request the patient records, and use our scripts to process the data. Or you can download the embedding encoded by MedCPT from link, and directly load using FAISS.

Expert-CoT and Expert-RAG

Filter (Left)

To guide the LLM reasoning and RAG retrieval based on question-specific expertise, we train a lightweight LLM-based filter to predict the key expertise attributes, including question’s subject area and certification level.

Expert-CoT (Mid)

Expert-CoT prompting guides the model’s reasoning by explicitly providing the subject area and certification level as starting point for the thought process.

Expert-RAG (Right)

The filter’s predicted subject area guides the retriever to search for relevant knowledge base entries and patient records tailored to the question’s subject area. The LLM then conditions on the predicted expertise and the retrieved documents to generate the final answer.

• Global: Retrieve the top M and N evidence documents from the entire KB and PR, respectively. This is a baseline.

• Filter then Retrieve (FTR): First filter the whole KB and PR to retain only documents matching the predicted subject area, then retrieve the top M and N documents from these filtered subsets

• Retrieve then Filter (RTF): First retrieve a larger candidate set from the whole KB and PR (e.g., 10 × Mfrom KB and 10 × N from PR), then filter out documents whose subject area do not match ˆsi, retaining the top M and N relevant documents

How to run the code

Run Benchmarks

Go benchmark folder

cd code/benchmark

We used Qwen/Qwen3-32B in our experiment to run our code,

• expert-cot prompting

  python benchmark_LLM.py 
      --model_name_or_path=Qwen/Qwen3-32B \
      --mode=infer
      --prompt=cot_attr
      --enable_think=True
  

Run Expert-RAG

Go EMSRAG folder

cd code/EMSRAG

We used Qwen/Qwen3-32B in the experiment to run our code,

• Expert-RAG-GT:

  python ems_rag.py 
      --model_name_or_path=Qwen/Qwen3-32B \
      --mode=infer \
      --prompt=cot_attr \
      --use_kb=True \
      --kb_k=32 \
      --use_pr=True \
      --pr_k=8 \
      --filter_mode=retrieve_then_filter \
      --use_adapter=False \
      --enable_think=True
  

  Retrieval mode: Filter then Retrieve (FTR)

  python ems_rag.py 
      --model_name_or_path=Qwen/Qwen3-32B \
      --mode=infer \
      --prompt=cot_attr \
      --use_kb=True \
      --kb_k=32 \
      --use_pr=True \
      --pr_k=8 \
      --filter_mode=filter_then_retrieve \
      --use_adapter=False \
      --enable_think=True
  

• Expert-RAG-Filter:

  Retrieval mode: Retrieve then Filter (RTF)

  python ems_rag.py 
      --model_name_or_path=Qwen/Qwen3-32B \
      --mode=infer \
      --prompt=cot_attr \
      --use_kb=True \
      --kb_k=32 \
      --use_pr=True \
      --pr_k=8 \
      --filter_mode=retrieve_then_filter \
      --use_adapter=True \
      --enable_think=True
  

  Retrieval mode: Filter then Retrieve (FTR)

  python ems_rag.py 
      --model_name_or_path=Qwen/Qwen3-32B \
      --mode=infer \
      --prompt=cot_attr \
      --use_kb=True \
      --kb_k=32 \
      --use_pr=True \
      --pr_k=8 \
      --filter_mode=filter_then_retrieve \
      --use_adapter=True \
      --enable_think=True
  

Code to crawl EMS-MCQA and KB

• EMS-MCQA (close source) check code/app-emtprep-com, you need to specify the username, and password in the script after subscribing the app.

Citations

If you use this dataset in your work, please consider citing our paper:

@misc{ge2025expertguidedpromptingretrievalaugmentedgeneration,
      title={Expert-Guided Prompting and Retrieval-Augmented Generation for Emergency Medical Service Question Answering}, 
      author={Xueren Ge and Sahil Murtaza and Anthony Cortez and Homa Alemzadeh},
      year={2025},
      eprint={2511.10900},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2511.10900}, 
}