MedicalQA

This is a medical question answering system with RAG on a medical knowledge graph.

Environment Requirements

CUDA 11.8
Python 3.12
PyTorch 2.5.1
streamlit 1.40.1
All the packages can be installed by conda env create -f environment.yaml.

Project Structure

MedicalQA
├── data
│    ├── kg_info
│    │    ├── fixed_entity_relation
│    │    │    ├── entity
│    │    │    │    └── ...
│    │    │    ├── relation
│    │    │    │    └── ...
│    │    │    └── ...
│    │    └── ...
│    └── match_info
│    │    └── ...
│    └── raw_data
│         └── ...
├── models
│    ├── Meta-Llama-3.1-8B-Instruct
│    │    └── ...
│    ├── Qwen2.5-14B-Instruct
│    │    └── ...
│    └── text2vec-base-chinese
│         └── ...
├── src
│    ├── knowledge_graph
│    │    ├── __init__.py
│    │    └── disease_to_cypher.py
│    ├── match
│    │    ├── __init__.py
│    │    ├── index_type.py
│    │    └── match_utils.py
│    ├── question_answer
│    │    ├── __init__.py
│    │    ├── generate.py
│    │    └── load.py
│    ├── spider
│    │    ├── chromedriver
│    │    ├── data_spyder.py
│    │    ├── paper_spider.py
│    │    ├── url_extract.py
│    │    └── other_all.py
│    └── ui
│         ├── __init__.py
│         ├── login.py
│         └── user_data_storage.py
├── tmp
│    └── user_credentials.json
├── main.py
├── logo.jpg
├── environment.yaml
├── LICENSE
└── README.md

Data Crawling

Instruction Papers Crawling

python src/spider/paper_spider.py (make sure chromedriver is in the same directory)

Structured Data Crawling

1. Crawl jbk.39.net
   python src/spider/url_extract.py
python src/spider/other_all.py
2. Crawl xywy.com
   python src/spider/data_spyder.py

Preprocess

python src/preprocess/convert_rawjson_to_line.py
python src/preprocess/old_new_merge.py
python src/preprocess/other_mecical_merge.py
These will generate 4 json files in data/raw_data.

Extract SPO

python src/preprocess/process.py
python src/preprocess/process_unstru.py
python src/kg/build_kg.py

Deduplicate

python src/encode/encode.py
python src/encode/delete.py

Build Graph

python src/kg/txt_to_kg.py

Run

streamlit run main.py

Demo

https://github.com/user-attachments/assets/f785cacf-d4de-47c2-8832-b8334e3af99a

Acknowledge

基于RAG与大模型技术的医疗问答系统