Medical_LLMs

Repository to act as a playground to test various LLMs in the context of the medical domain

SemEval2023

Set Compostions

All sets are balanced in labels, having 50% of Entailment and Contradiction

---

T5 Results

In order to obtain these results, I used Flan-T5 from huggingface, and used the following generation prompt: $premise \n Question: Does this imply that $hypothesis? $options, checking the outputs for "Entailment" or "Contradiction".

Train Set (0-shot)

Dev Set (0-shot)

Dev Set (fine-tuned)

---

qCammel family Results

In order to use the Clinical Cammel models, we will obtain them from huggingface hub, as it very easily allows us to quantize and test different model sizes.

Prompts

Alpaca Template (_alp + {prompt_used}): Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n$task_prompt\n### Response:

Asclepius Template (_asclp + {prompt_used}): You are an intelligent clinical language model.\nBelow is a snippet of clinical trial data and a following instruction from a healthcare professional. Write a response that appropriately completes the instruction. The response should provide the accurate answer to the instruction, while being concise.\n\n[Instruction Begin]\n{prompt}\n[Instruction End]

Base T5 Prompt (t5): $premise \n Question: Does this imply that $hypothesis? $options

Perplexity

Instead of trying to generate "Entailment" and "Contradiction", it's also possible to analize the output probabilities of the tokens, not requiring the model to explicity generate those tokens.

We will denote these differences by using _gen when the full generation is processed, and _perp when the perplexity score is used.

Model Links

Asclepius-Llama2-13B

Asclepius-13B-GPTQ

qCammel-13-GPTQ

qCammel-13B-Combined-Data-GPTQ

qCammel-13B-Role-Playing-GPTQ

qCammel-70-x-GPTQ

qCammel-70-x-GPTQ-gptq-4bit-128g

qCammel-70-x-GPTQ-gptq-3bit-128g

---

Dev Set (0-shot)

Testing different task prompts (with apl{t5} + gen)

"1" : "$premise \n Based on the paragraph above can we conclude that $hypothesis? $options",

"2" : "$premise \n Based on that paragraph can we conclude that this sentence is true? $hypothesis $options",

"3" : "$premise \n Can we draw the following conclusion? $hypothesis $options",

"4" : "$premise \n Does this next sentence follow, given the preceding text? $hypothesis $options",

"5" : "$premise \n Can we infer the following? $hypothesis $options",

"6" : "Read the following paragraph and determine if the hypothesis is true: $premise \n Hypothesis: $hypothesis \n $options",

"7" : "Read the text and determine if the sentence is true: $premise \n Hypothesis: $hypothesis \n $options",

"8" : "Can we draw the following hypothesis from the context? Context: $premise \n Hypothesis: $hypothesis \n $options",

"9" : "Determine if the sentence is true based on the text below: $hypothesis \n $premise \n $options",

"10" : "$premise \n Question: Does this imply that $hypothesis? $options"

---

Using Alpaca Template (_alp) + T5 prompt yes/no (t5) + generation (gen)

Prompt: Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n$premise \n Question: Does this imply that $hypothesis?\n Respond by outputting Yes or No. Be as accurate as possible. \n\n\n### Response:

Using Alpaca Template (_alp) + T5 prompt entailment/contradiction (t5) + generation (gen)

Prompt: Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n$premise \n Question: Does this imply that $hypothesis?\n Respond by outputting Entailment or Contradiction. Be as accurate as possible. \n\n\n### Response:

---

Using Alpaca Template (_alp) + T5 prompt yes/no (t5) + generation with CoT (gen-CoT)

Prompt: Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n$premise \n Question: Does this imply that $hypothesis?\n Let's think step by step, and end by outputting Yes or No. Be as accurate as possible. \n\n\n### Response:

---

New Prompts Generation

"1" : "$premise \n Based on the paragraph above can we conclude that $hypothesis? $options", "3" : "$premise \n Can we draw the following conclusion? $hypothesis $options", "5" : "$premise \n Can we infer the following? $hypothesis $options", "8" : "Can we draw the following hypothesis from the context? Context: $premise \n Hypothesis: $hypothesis \n $options", "10" : "$premise \n Question: Does this imply that $hypothesis? $options"

Using these base prompts and gpt3.5-turbo, I generated the following prompts: "1_3" : "$premise Can we draw a reasonable conclusion regarding $hypothesis? Consider the $options", "1_5" : "$premise Can we draw a conclusion regarding $hypothesis based on the provided information? Consider the $options", "1_8" : "$premise Can we derive a conclusion regarding $hypothesis from the information provided? Consider the $options", "1_10" : "$premise \n Given the context, is it reasonable to assume that $hypothesis? $options",

"3_1" : "$premise Considering the information provided, can we draw the conclusion that $hypothesis? $options", "3_5" : "$premise \n Can we deduce the following? $hypothesis $options", "3_8" : "Can we infer the following hypothesis based on the given $premise? Hypothesis: $hypothesis $options", "3_10" : "$premise Assuming $hypothesis, what are the $options to consider?",

"5_1" : "Can we infer whether $premise supports the hypothesis that $hypothesis? $options ", "5_3" : "$premise Can we deduce the following outcome? $hypothesis $options", "5_8" : "Can we draw a plausible hypothesis based on the given context? Context: $premise Hypothesis: $hypothesis $options", "5_10" : "$premise \n Question: Can we infer the following? $hypothesis $options ",

"8_1" : "Can we draw a plausible hypothesis from the given context? Context: $premise Hypothesis: $hypothesis $options", "8_3" : "Can we infer the following outcome from the given scenario? Scenario: $premise \n Outcome: $hypothesis \n $options", "8_5" : "Can we derive the following hypothesis based on the given $premise? Furthermore, can we deduce the validity of $hypothesis considering $options?", "8_10" : "Considering the context outlined in $premise, can we deduce $hypothesis? Evaluate the following options: $options",

"10_1" : "Given the information provided, is it reasonable to infer that $hypothesis based on the $premise? $options", "10_3" : "$premise \n Can we infer that $hypothesis? $options", "10_5" : "$premise \n Can we deduce the following? $hypothesis $options", "10_8" : "Given the context: $premise Is it reasonable to conclude: $hypothesis?$options"

---

LLaMa-2-chat-hf

---

Other notes

Max token limit per model:

NVIDIA A100-PCIE-40GB

[qCammel-13-GPTQ] - 4096 tokens [qCammel-70-x-GPTQ] - 2600 tokens (Can use more tokens than 4-bit because doesn't use group size) [qCammel-70-x-GPTQ-4bit-32g] - 1250 tokens [qCammel-70-x-GPTQ-4bit-64g] - 1650 tokens [qCammel-70-x-GPTQ-4bit-128g] - 2050 tokens [qCammel-70-x-GPTQ-3bit-1g] - 4096 tokens [qCammel-70-x-GPTQ-3bit-128g] - 4096 tokens