K2MnO4/cognition-corrector

stars

106

commits

Python

primary language

Nov 24, 2024

updated

README

cognition-corrector

The Overview of Cognition-Corrector of Medical GQA System

Introduction

This is a congnition corrector for mitigating the hallucination in medical field. The method gradually enhances the factuality, consistency of generated answers through knowledge acquisition and consistency feedback.

Structure

Core: baseline and cognition-corrector codes.
Data Process:
- convert the original dataset into specific format.
- responses generated by baseline and corrector.
Evaluate: evaluation metrics tools.
templates: prompt templates.
visualize: result visualization.

Environment Setting

conda create -n nlpEnv python=3.9.20
conda activate nlpEnv
pip install -r requirements.txt

Results

We run our baselines and correctors to generate candidate answers on one GPU: Nvidia GeForce RTX 3090. Besides, we also used the same device to evaluate our results.

1. PubMedQA

Model	F1	ROUGE-L	NLI-Samp	NLI-Sent
Gpt2-Large-Medical	0.140	0.138	0.020	0.120
Gpt2-Large-Medical_C	0.113	0.118	0.184	0.244
Alpaca-LoRA	0.223	0.171	0.200	0.237
Alpaca-LoRA_C	0.236	0.204	0.460	0.574
MedAlpaca-LoRA	0.196	0.165	0.258	0.179
MedAlpaca-LoRA_C	0.218	0.183	0.525	0.521

2. MediQA2019

Model	F1	ROUGE-L	NLI-Samp	NLI-Sent
Gpt2-Large-Medical	0.165	0.141	0.427	0.267
Gpt2-Large-Medical_C	0.138	0.121	0.490	0.290
Alpaca-LoRA	0.119	0.118	0.673	0.584
Alpaca-LoRA_C	0.158	0.150	0.740	0.605
MedAlpaca-LoRA	0.114	0.106	0.673	0.587
MedAlpaca-LoRA_C	0.116	0.108	0.580	0.416

3. MeshQA

Model	F1	ROUGE-L	NLI-Samp	NLI-Sent
Gpt2-Large-Medical	0.151	0.146	0.360	0.271
Gpt2-Large-Medical_C	0.144	0.139	0.563	0.339
Alpaca-LoRA	0.211	0.184	0.780	0.655
Alpaca-LoRA_C	0.214	0.187	0.780	0.712
MedAlpaca-LoRA	0.185	0.162	0.710	0.653
MedAlpaca-LoRA_C	0.162	0.145	0.730	0.693

4. LiveQAMed2017

Model	F1	ROUGE-L	NLI-Samp	NLI-Sent
Gpt2-Large-Medical	0.180	0.151	0.327	0.219
Gpt2-Large-Medical_C	0.153	0.141	0.475	0.232
Alpaca-LoRA	0.137	0.138	0.404	0.346
Alpaca-LoRA_C	0.190	0.168	0.577	0.422
MedAlpaca-LoRA	0.134	0.133	0.390	0.335
MedAlpaca-LoRA_C	0.130	0.130	0.450	0.384

Contributors

15989488337

61 commits

K2MnO4

43 commits

lzh-um

2 commits

Similar repos

ziweiji/Self_Reflection_Medical

Code for paper Towards Mitigating LLM Hallucination via Self Reflection

dmis-lab/OLAPH

OLAPH: Improving Factuality in Biomedical Long-form Question Answering

YuSun-AI/ReasonMed

Dataset

FreedomIntelligence/Awesome-AI4Med

A curated list of medical LLMs, multimodal systems, datasets, benchmarks, and more. 🏥

2,877

lingshu-medical-mllm/Lingshu-7B

Model

lingshu-medical-mllm/Lingshu-32B

Model

m42-health/Llama3-Med42-8B

Model

m42-health/Llama3-Med42-70B

Model

Languages

Python

100.0%

K2MnO4/cognition-corrector

stars

106

commits

Python

primary language

Nov 24, 2024

updated

README

cognition-corrector

The Overview of Cognition-Corrector of Medical GQA System

Introduction

Structure

Core: baseline and cognition-corrector codes.
Data Process:
- convert the original dataset into specific format.
- responses generated by baseline and corrector.
Evaluate: evaluation metrics tools.
templates: prompt templates.
visualize: result visualization.

Environment Setting

conda create -n nlpEnv python=3.9.20
conda activate nlpEnv
pip install -r requirements.txt

Results

We run our baselines and correctors to generate candidate answers on one GPU: Nvidia GeForce RTX 3090. Besides, we also used the same device to evaluate our results.

1. PubMedQA

Model	F1	ROUGE-L	NLI-Samp	NLI-Sent
Gpt2-Large-Medical	0.140	0.138	0.020	0.120
Gpt2-Large-Medical_C	0.113	0.118	0.184	0.244
Alpaca-LoRA	0.223	0.171	0.200	0.237
Alpaca-LoRA_C	0.236	0.204	0.460	0.574
MedAlpaca-LoRA	0.196	0.165	0.258	0.179
MedAlpaca-LoRA_C	0.218	0.183	0.525	0.521

2. MediQA2019

Model	F1	ROUGE-L	NLI-Samp	NLI-Sent
Gpt2-Large-Medical	0.165	0.141	0.427	0.267
Gpt2-Large-Medical_C	0.138	0.121	0.490	0.290
Alpaca-LoRA	0.119	0.118	0.673	0.584
Alpaca-LoRA_C	0.158	0.150	0.740	0.605
MedAlpaca-LoRA	0.114	0.106	0.673	0.587
MedAlpaca-LoRA_C	0.116	0.108	0.580	0.416

3. MeshQA

Model	F1	ROUGE-L	NLI-Samp	NLI-Sent
Gpt2-Large-Medical	0.151	0.146	0.360	0.271
Gpt2-Large-Medical_C	0.144	0.139	0.563	0.339
Alpaca-LoRA	0.211	0.184	0.780	0.655
Alpaca-LoRA_C	0.214	0.187	0.780	0.712
MedAlpaca-LoRA	0.185	0.162	0.710	0.653
MedAlpaca-LoRA_C	0.162	0.145	0.730	0.693

4. LiveQAMed2017

Model	F1	ROUGE-L	NLI-Samp	NLI-Sent
Gpt2-Large-Medical	0.180	0.151	0.327	0.219
Gpt2-Large-Medical_C	0.153	0.141	0.475	0.232
Alpaca-LoRA	0.137	0.138	0.404	0.346
Alpaca-LoRA_C	0.190	0.168	0.577	0.422
MedAlpaca-LoRA	0.134	0.133	0.390	0.335
MedAlpaca-LoRA_C	0.130	0.130	0.450	0.384