Knowledge Retrieval Based on Generative AI

Architecture: RAG (Retrival Augmented Generation)

Description

The purpose of this study is to conduct the evaluation process of knowledge retrieval through the Generative AI model.

---

Figure：Basic Flow

---

Figure: Obtain information from wikipedia and establish a knowledge base (the knowledge base has been established in advance)

---

Figure: Using the contents of the knowledge base to build vector indexes

---

Figure: Using the dense vector retrieval to retrieve all content in the knowledge base that is similar to the query (question)

---

Figure: Re-rank the previously obtained content and corresponding document IDs to obtain the most relevant content

---

Figure: Integrate the search results into a customized User Prompt and make inferences through the Generative AI model

---

圖：The Flow of Comparison

---

圖：The methods of Comparison

Build environment using Conda

conda create --name adl python=3.10
conda activate adl

Packages

# Step 1
pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121

# Step 2
pip install flash-attn --no-build-isolation

# Step 3
conda install -c pytorch -c nvidia faiss-gpu=1.9.0

# Step 4
pip install -r requirements.txt

Notice:

• Please use GPUs that supports flash attention, such as the Ampere or Ada series, install additional flash-attn, or set use_flash_attention to False in web_api_gemma.py.

The Flow of Execution

1. Change directory to knowledge_retrieval_based_on_generative_ai:

   cd knowledge_retrieval_based_on_generative_ai
   

2. Download Vector Index and Knowledge Base

   bash ./download.sh
   

3. Start the Web API, taking gemma as an example (it is recommended to use tmux or screen for background execution)

   # Using tmux for background execution
   tmux new -s web_api
   
   # After establishing the tmux session, start the Web API
   cd server
   python web_api_gemma.py --llm_name=google/gemma-2b-it --tokenizer_use_cuda_id=0 --max_new_tokens=30 --model_max_length=8192
   
   # (After successful execution)
   # Ctrl + B, release it and press D immediately to leave the tmux session and let it execute in the background.
   # To re-enter the session, enter: tmux a -t web_api
   

   Notice:

   • The first execution will download the model, which will take some time; some models require Hugging Face to confirm authorization, please obtain authorization before executing.
   • The parameter values ​​of --llm_name in web_api_gemma.py and run_inference.py need to be consistent.

4. After the Web API is started, execute the inference program:

• After inference, the final_predicted_result_file.json file will be generated for evaluation.
• The json file will include the inference results of "not using RAG" and "using RAG".

  time python run_inference.py --llm_name=google/gemma-2b-it --task_alias=TTQA
  

5. After running the inference, evaluate:

• After the evaluation is completed, the results folder will be automatically created and the evaluation results will be stored in it.
• The evaluation results are divided into ...rag... and ...no-rag..., and the evaluation results will be placed inside.

  time python run_calculation.py --task_alias=TTQA
  
  # If it is a TMMLU-PLUS task, execute:
  # time python run_calculation.py --task_alias=TMMLU-PLUS
  

Optional

Other references to --llm_name are as follows:

# To modify the parameter value during inference, please modify it together with the `--llm_name` parameter value of Web API. Both sides must be synchronized (exactly the same)
# Please select the corresponding `--llm_name` according to the model architecture of server/web_api_xxx.py.
# For example, yentinglin/Taiwan-LLM-7B-v2.0.1-chat corresponds to web_api_twllama.py
--llm_name=yentinglin/Taiwan-LLaMa-v1.0
--llm_name=yentinglin/Taiwan-LLM-7B-v2.0.1-chat
--llm_name=yentinglin/Taiwan-LLM-13B-v2.0-chat
--llm_name=meta-llama/Llama-2-7b-chat-hf
--llm_name=meta-llama/Llama-2-13b-chat-hf
--llm_name=meta-llama/Llama-2-70b-chat-hf
--llm_name=mistralai/Mixtral-8x7B-Instruct-v0.1
--llm_name=yentinglin/Taiwan-LLM-8x7B-DPO
--llm_name=INX-TEXT/Bailong-instruct-7B
--llm_name=MediaTek-Research/Breeze-7B-Instruct-v0_1
--llm_name=MediaTek-Research/Breexe-8x7B-Instruct-v0_1
--llm_name=google/gemma-7b-it
--llm_name=google/gemma-2b-it

Other task aliases --task_alias are as follows:

--task_alias=TTQA
--task_alias=TMMLU-PLUS