LLM Chat Bot

This project implements a QA chatbot system that generates easy-to-understand (ELI5-style) answers by combining LLAMA-3.2-1B’s structured output with BART-Large-CNN’s summarization capabilities. The system includes a comprehensive quality assessment pipeline to ensure answer relevance and clarity.

---

Requirements

• Python 3.9 or above
• pipenv for managing dependencies

---

Installation

1. Clone the repository:

   git clone [email protected]:Khanh04/capstone.git
   cd capstone
   

2. Install dependencies:

   Make sure pipenv is installed:

   pip install pipenv
   

   Then, install the project dependencies:

   pipenv install
   

3. Activate the virtual environment:

   pipenv shell
   

System Pipeline

Step 0: Finetune LLAMA and GPT2 Models

• Scripts: train_llama, train_gpt2
• Input: ELI5 dataset (260,380 QA pairs)
• Output: llama_finetuned_model, gpt2_finetuned_model

---

Step 1: Generate 100 QA Pairs (100 Questions, 500 Answers)

• Script: get_QA_pairs
• Input: Test set of 100 questions
• Output: QA_pairs.csv

Dataset Answer Generation:

1. LLAMA Structured Answer
2. LLAMA Finetuned
3. GPT2 Answer
4. GPT2 Finetuned

---

Step 2: Data Collection for Answer Quality Assessment Model Building

2-1

• No Script Required
• Output: qa_pairs.csv
  • Random 1000 questions, 1000 answers (good answers generated by Claude)
  • Test set:
    • 100 GPT2 answers (all bad; labeled as good/bad by Claude)
    • 100 GPT2 finetuned answers (all bad; labeled as good/bad by Claude)
    • 100 dataset answers (66 good: 34 bad; labeled as good/bad by Claude)

2-2

• Script: llama_gen_bad_get_qa_pairs2
• Output: qa_pairs2.csv
  • Random 1200 questions, 1200 LLAMA finetuned answers
  • Labeled:
    • Good: ROUGE_L > 0.651
    • Bad: ROUGE_L ≤ 0.651

2-3

• Scripts: llama_gen_bart_get_qa_pairs3_simple
• Output:
  • qa_pairs3.csv (intermediate file)
  • qa_pairs3_simple.csv
  • Random 247 questions, 247 LLAMA + BART-Large-CNN answers (233 good: 14 bad; labeled good/bad by Claude)

---

Step 3: Build Answer Quality Assessment Model

• Script: qa_pairs_scoring_fit
• Input: qa_pairs.csv, qa_pairs2.csv, qa_pairs3_simple.csv
• Output: qa_model (Answer Quality Assessment Model)

---

Step 4: Answer Quality Assessment for Test Questions/Answers

4-1: Generate and Assess Answers

• Script: llama_gen_bart_get_QA_pairs
• Output: QA_pairs_simple.csv
  • Existing answers:
    • 100 LLAMA finetuned answers
    • 100 LLAMA Structured + BART-Large-CNN summary answers
    • 100 LLAMA Structured + LLAMA Summary answers

4-2: Score Answers Using the Quality Assessment Model

• Script: QA_pairs_scoring_byfittedmodel
• Input: QA_pairs_simple.csv, qa_model
• Output: QA_pairs_scoring_byfittedmodel.csv

Extra Step: Alternative Answer Quality Assessment

• Script: QA_pairs_scoring_st_ce
• Input: QA_pairs.csv
• Output: QA_pairs_scoring_st_ce.csv

---

Key Findings

1. Limitations of Fine-Tuned Models:

   • LLAMA: Repetitive answers.
   • GPT2: Often produced meaningless content.

2. Answer Quality Assessment Model:

   • Built using ROBERTa-base.
   • Effectively checks relevance between questions and answers.

3. Best Results Achieved By:

   • LLAMA + BART-Large-CNN combination.
   • Alternative: LLAMA with self-summarization.

---

Repository Structure

├── main.py
├── notebooks/
├── data/

---

Future Work

• Enhance the summarization model with dynamic context selection.
• Experiment with larger datasets to improve fine-tuned model performance.
• Incorporate multimodal data (e.g., images and text) for richer QA capabilities.

License

This project is licensed under the MIT License. See the LICENSE file for details.

---