CPU-Assisted Language Model (LM) Serving

Install Prerequisites

sudo apt install -y docker.io
sudo usermod -aG docker $USER
newgrp docker

On the GPU machine install also the NVIDIA container toolkit

sudo apt install -y nvidia-container-toolkit

GPU Machine (Backend) Setup

Edit properties.json

Manually edit the etc/properties.json file if needed to use the port you need.

Build GVirtuS image on the GPU machine

make docker-build-gvirtus

Run the GVirtuS backend daemon on the GPU machine

make run-gvirtus-backend

In a new terminal attach to the GVirtuS backend container

make attach-gvirtus-backend

Install the required python packages

/fastapi/setup.sh

Edit server.py

Edit fastapi/server.py line 20 to serve the model(s) and API endpoint of your choice. Optionally, change the model you want for each task.

nano /fastapi/server.py

Run the LM inference server using native GPU at your desired port

/fastapi/deploy_native <port>

Non-GPU Machine (Frontend) Setup

Edit properties.json

Manually edit the etc/properties.json file if needed to set the IP and port of your GVirtuS backend.

Build GVirtuS image on the GPU machine

make docker-build-gvirtus

Run the GVirtuS frontend on the non-GPU machine

make run-gvirtus-frontend

Install the required python packages

/fastapi/setup.sh

Run the LM inference server through GVirtuS at your desired port

/fastapi/deploy_gvirtus <port>

Repeat this process for as many non-GPU machines you want to add

Query sender setup

Edit sender.py

Edit fastapi/sender.py lines 17-23 to reflect your machines' IP and set up which one is the GPU machine. Also, edit line 44 to select the models of your choice. Optionally, edit other parameters if you need to.

nano /fastapi/sender.py

Send the queries

/fastapi/query.sh