About The Project

This project aims to generate synthetic data for translating English quotes into Vietnamese using the Gemini API. The generated data serves as the foundation for fine-tuning the Gemma-2b model, enhancing its ability to produce high-quality Vietnamese quotes. The ultimate goal is to deploy the model and make it production-ready and highly scalable using Kubernetes on Google Cloud Platform (GKE) and integrate with CI-CD pipeline. This is my first project of MLOps course 1 at FSDS. I'd like to give a big credit to my instructor Mr. Quan Dang for his guidance and support.

(back to top)

Built With

• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 

(back to top)

Getting Started

If you don't want to fine-tune the model, you can skip the data generation section and training section and move ahead to the deployment section. Checkout my data and model on the Huggingface hub as prequisites for the deployment phase.:

• Dataset: dinhlnd1610/Vietnamese_Quote_Dataset_100K
• Model: dinhlnd1610/gemma-2b-quote-generation-82000

General Setup

1. Clone the repository

   git clone https://github.com/dinhln03/LLM_Quote_Genarator.git
   

2. Cd into the project directory

   cd LLM_Quote_Genarator
   

3. Run export ROOT_DIR=$(pwd) for easier nagivation

I. Data Generation

1. Setup

   • Install conda for managing Python environments
   • Install gdown

2. cd $ROOT_DIR/training/translate
   

3. Create new .env file based on .env.example at $ROOT_DIR/training/translate path and populate the variables there

   set -a && source .env && set +a
   

4. Download English quote dataset and install the required packages

   gdown https://drive.google.com/uc?id=11MmVMc0khvB94W0zqDyUHowEEG650BSX
   conda create -n training python=3.10 -y && conda activate training  && pip install -r requirements.txt
   

5. Run the Script to generate Vietnamese quotes. Data will be saved in data folder and log will be saved in logs folder

   python main.py main \
     --data_path=$data_path \
     --sample_length=$sample_length \
     --start_batch=$start_batch \
     --end_batch=$end_batch \
     --num_thread=$num_thread
   

   # Example
   python main.py main --data_path=./data.csv --sample_length=20000 --start_batch=0 --end_batch=4000 --num_thread=5
   

   Parameter	Description
   $data_path	Path to the downloaded English quote dataset
   $sample_length	Number of quotes to translate
   $start_batch	Starting batch number
   $end_batch	Ending batch number
   $num_thread	Number of threads to use

   Key Notes:

   • You should choose num_thread such that (end_batch - start_batch + 1) is divisible by num_thread.
   • Calculate num_thread using:

     num_thread = (end_batch - start_batch) // 5
     

6. Merge json data in folder data to a single json file

   python utils/merge_file.py 
   

7. Checkout the notebook to do some preprocessing data step such as: removing duplicate data point, remove toxic words,.. and push to huggingface hub. You can include more preprocessing steps such as removing more toxic words, filtering by perplexity,...

(back to top)

II. Fine-tuning Gemma-2b model

1. Install the required packages

   cd $ROOT_DIR/training/training_scripts
   sh run.sh
   

2. Download google/gemma-2b model on huggingface hub and write appropriate chat template for generating quotes

   cd utils
   python training_utils.py download_model google/gemma-2b ../../gemma-2b
   

3. Fine-tune gemma-2b model on translated data produced in the previous step

   cd $ROOT_DIR/training/training_scripts
   

   If you have only 1 GPU, just run

   export CUDA_VISIBLE_DEVICES=0
   python training.py
   

   If you have 1 node and multiple GPUs, run Distributed Data Parallel

   export CUDA_VISIBLE_DEVICES=1,2,...
   torchrun --standalone --nnodes=1 --nproc-per-node=$NUM_GPUs training.py
   

4. Login to Wandb to monitor the training process

5. Merge the lora adapter with the base model.

   cd utils
   python training_utils.py merge_model \
      --base_model_path=$base_model_path \
      --lora_path=$lora_model_path \
      --output_path=$output_model_path
   

   Parameter	Description
   $base_model_path	Path to the orginal google/gemma2b model
   $lora_model_path	Path to the fine-tuned lora adapter
   $output_model_path	Path to save the merged model

6. Checkout the notebook to push the model to huggingface hub.

(back to top)

III. Deployment

1. Setup

   • Install kubectl to communicate with the Kubernetes API server
   • Install kubectx and kubens for easier navigation between clusters and namespaces
   • Install helm to manage templating and deployment of Kubernetes resources
   • Install yq to edit YAML files

     sudo wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/bin/yq && sudo chmod +x /usr/bin/yq && hash -r
     

   • Create a new .env file based on .env.example and populate the variables there

     cd $ROOT_DIR
     set -a && source .env && set +a
     

2. Deploy model on RunPod

   Note: You need to have a RunPod account and push the model to the huggingface hub first

   • Log in and click on Serverless tab

   • In the Serverless vLLM tab, choose start

   • Config path to your model and other configurations properly

   • Follow this tutorial to get $RUNPOD_ENDPOINT_ID, $RUNPOD_API_KEY and $HF_MODEL_NAME and populate them into app/constants.py file

3. • Containerize the FastAPI app using Docker:

   docker build -t quote-generator:lastest -f Dockerfile .
   

   • Quick test the container:

   docker run -p 30000:30000 quote-generator:lastest
   

   • Access the FastAPI app at http://localhost:30000/docs and test the API

   • Rename the image to your Docker Hub username and push it to Docker Hub (remenber to login to Docker Hub first):s

   docker tag quote-generator:lastest$DOCKER_USERNAME/quote-generator:lastest
   docker push $DOCKER_USERNAME/quote-generator:lastest
   

   Cool! Now you have a FastAPI app running in a container. Let's "go to the moon" and deploy it to the cloud with K8s.

4. Creating GKE cluster using Terraform

   • Login to GCP console and create new project

   • Input your project ID in in variable "project_id" in terraform/variables.tf

     

   • Login to GCP using gcloud CLI

     gcloud auth application-default login
     

   • Provision a new GKE cluster using Terraform

     cd $ROOT_DIR/iac/terraform
     terraform init
     terraform plan
     terraform apply
     

     

   • Connect to the GKE cluster

     gcloud container clusters get-credentials $CLUSTER_NAME --region $REGION --project $PROJECT_ID
     

   That's it! You have a GKE cluster up and running. In case you want to delete the cluster, just run terraform destroy.

5. Deploy the FastAPI app to GKE (remember to change the image in the quote_gen_chart/values.yaml file to your Docker Hub username)

   • Switch context to the GKE cluster

     kubectx gke_${PROJECT_ID}_${REGION}_${CLUSTER_NAME}
     

   • Install nginx-ingress controller

     cd $ROOT_DIR/helm/nginx-ingress
     helm upgrade --install nginx-ingress . --namespace nginx-ingress --create-namespace
     

   • Get the external IP of the nginx-ingress controller

     export EXTERNAL_IP=$(kubectl get svc -n nginx-ingress | grep "LoadBalancer" | awk '{print $4}')
     

     
   • Create Host for the external IP and export it as env variable. Here, I use nip.io for dynamic DNS ( You can use your own domain or append new host to /etc/hosts that point to the external IP)

     export HOST=app.$EXTERNAL_IP.nip.io
     

   • Install quote-generator appplication

     cd $ROOT_DIR/helm/quote_gen_chart
     yq e '.host = env(HOST)' -i values.yaml
     helm upgrade --install quote-generator . --namespace quote-gen --create-namespace
     

   • Now can can access the FastAPI swagger UI at <HOST>/docs

(back to top)

IV. Observability

So far, we have deployed the model and the FastAPI app to GKE. Now, we need to monitor the performance of the model and the app. We will use Prometheus and Grafana for monitoring the model and the app, Jaeger for tracing the requests and Elastic Search and Kibana for collecting system logs. Let's cook !

1. Increase inotify watch limits to Kubernetes instances.

   cd $ROOT_DIR/observability/inotify
   kubectl apply -f inotify-limits.yaml
   

2. Create a separate namespace for observability and switch to it

   kubectl create namespace observability
   kubens observability
   

3. Install Jaeger for tracing application

   cd $ROOT_DIR/observability/jaeger
   yq e '.query.ingress.hosts = [env(HOST)]' -i values.yaml
   helm upgrade --install jaeger . --namespace observability 
   

4. Install ELK stack for collecting logs

   # Intall ECK operator
   cd $ROOT_DIR/observability/elasticcloud/deploy/eck-operator
   kubectl delete -f https://download.elastic.co/downloads/eck/2.13.0/crds.yaml
   kubectl create -f https://download.elastic.co/downloads/eck/2.13.0/crds.yaml
   kubectl apply -f https://download.elastic.co/downloads/eck/2.13.0/operator.yaml
   #Install ELK stack
   cd $ROOT_DIR/observability/elasticcloud/deploy/eck-stack
   kubectl get serviceaccount filebeat -n elk &> /dev/null && kubectl delete serviceaccount filebeat -n elk || true
   kubectl get clusterrolebinding filebeat -n elk &> /dev/null && kubectl delete clusterrolebinding filebeat -n elk || true
   kubectl get clusterrole filebeat -n elk &> /dev/null && kubectl delete clusterrole filebeat -n elk || true
   helm upgrade --install elk -f values.yaml .
   

5. Install Prometheus + Grafana for monitoring system

   cd $ROOT_DIR/observability/metric
   yq e '.data."config.yml" |= sub("webhook_url: .*", "webhook_url: env(DISCORD_WEBHOOK_URL)")' -i charts/alertmanager/templates/configmap.yaml 
   helm upgrade --install prom-graf . --namespace observability
   

6. Access the monitoring tools:

   • Jaeger: <HOST>/tracing-jaeger

   • Kibana:

     nohup kubectl port-forward -n observability svc/elk-eck-kibana-kb-http 5601:5601 > /dev/null 2>&1 &
     

     Access Kibana at http://localhost:5601

     Get the password for the elastic user by running:

     kubectl get secret elasticsearch-es-elastic-user -n observability -o jsonpath='{.data.elastic}' | base64 -d
     

     Login and check for logs

   • Grafana:

     nohup kubectl port-forward -n observability svc/grafana 3000:3000 > /dev/null 2>&1 &
     

     Access Grafana at http://localhost:3000

     Login with username: admin and password: admin and check for metrics

   • Alertmanager: You will receive alerts in your Discord channel

(back to top)

V. CI-CD Pipeline

So far, we have deployed the model and the FastAPI app to GKE, monitored the performance of the model and the app. Now, we need to automate the deployment process using Jenkins. Let's do it !

1. Setup

   • Rebuild Jenskin image with docker and helm installed

     cd $ROOT_DIR/custom_images/jenkins
     docker build -t $DOCKER_USERNAME/jenkins:lts -f Dockerfile .
     

   • Push the image to Docker Hub (remember to login to Docker Hub first)

     docker push $DOCKER_USERNAME/jenkins:lts
     

2. Build up a VM instance on GCP using Ansible

   • Install pre-requisites

     cd $ROOT_DIR/iac/ansible/deploy_jenkins
     pip install -r vm_requirements.txt
     

   • Create a Service Account on GCP and download the JSON key file. Then put the file in the iac/ansible/secrets folder

   • Build up the VM instance

     ansible-playbook deploy_jenkins.yml --extra-vars "project=$PROJECT_ID service_account_file=$secret_file_path"
     

     For example: ansible-playbook create_compute_instance.yaml --extra-vars "project=quote-generation-442617 service_account_file=../secrets/quote-generation-442617-f1139970a9ba.json"

     

     You have successfully created a VM instance on GCP. Now, let's install Jenkins on it.

3. Install Jenkins on VM

   • Create a new ssh key pair

     mkdir $ROOT_DIR/iac/ansible/.ssh
     ssh-keygen -f $ROOT_DIR/iac/ansible/.ssh/jenkins 
     

   • Cat the public key and add it to the VM instance

     cat $ROOT_DIR/iac/ansible/.ssh/jenkins.pub
     

   • Get your VM's external IP address and add it to the iac/ansible/inventory file
   • Install Jenkins on the VM

     ansible-playbook -i ../inventory deploy_jenkins.yaml
     

     

4. Access Jenkins at http://<VM_IP_ADDR>:8081 and find password by running

   ssh -i $ROOT_DIR/iac/ansible/.ssh/jenkins <username>@<VM_IP_ADDR>    # Example: ssh -i $ROOT_DIR/iac/ansible/.ssh/jenkins dinhln@35.240.163.52
   sudo docker exec -it jenkins cat /var/jenkins_home/secrets/initialAdminPassword
   ctrl + D  # to exit the ssh session
   

   

5. Install the recommended plugins and create an admin user

6. Choose Skip and continue as admin and Save and finish

7. Choose Start using Jenkins

8. We will be redirected to the Jenkins dashboard

9. Choosing manage Jenkins -> Manage plugins -> Available -> Search for Docker and Kubernetes and install them

10. Connect Jenkins to GKE cluster

    • Choose Manage Jenkins -> Cloud -> New Cloud
    • Get cluster ip and put it in the Kubernetes URL field

      current_context=$(kubectl config current-context)
      current_cluster=$(kubectl config view -o jsonpath="{.contexts[?(@.name=='$current_context')].context.cluster}")
      kubectl config view -o jsonpath="{.clusters[?(@.name=='$current_cluster')].cluster.server}"
      

    • Get the server certificate and put it in the Kubernetes server certificate key field

      current_context=$(yq e '.current-context' ~/.kube/config)\
      current_cluster=$(yq e ".contexts[] | select(.name == \"$current_context\") | .context.cluster" ~/.kube/config)\
      yq e ".clusters[] | select(.name == \"$current_cluster\") | .cluster.\"certificate-authority-data\"" ~/.kube/config
      

    • Grants admin privileges to the "default" service account in that quote-gen namespace, and also grants admin privileges to anonymous users within the same namespace.

      kubectl create clusterrolebinding model-serving-admin-binding \
      --clusterrole=admin \
      --serviceaccount=model-serving:default \
      --namespace=quote-gen
      
      kubectl create clusterrolebinding anonymous-admin-binding \
      --clusterrole=admin \
      --user=system:anonymous \
      --namespace=quote-gen
      

    • Test connection

11. Create a new pipeline

• Choose New Item -> Multi Branch Pipeline -> OK

  

• Configure the pipeline

  

  

• Add Docker Hub credentials

  

  

  

  

11. Finally, add new webhook to your Github Reposistory

• Go to your Github repository -> Settings -> Webhooks -> Add webhook
• Add the Jenkins URL and append /github-webhook/ to it
• Choose Push and Pull Request and Active the webhook
• Save the webhook 12.Demo

(back to top)