Quick VLLM

Welcome to quick VLLM, a slim Python wrapper that makes talking to a vLLM or LM Studio server easier.

---

Features

---

Installation

pip install --upgrade vllm
pip install git+https://github.com/SamBlouir/quick_vllm.git

macOS

On macOS the default multiprocessing start method is spawn. quick_vllm now automatically switches to fork so that pooling works correctly.

---

Starting a vLLM server (example)

vllm serve \
  meta-llama/Llama-3.2-1B-Instruct \
  --tokenizer meta-llama/Llama-3.2-1B-Instruct \
  --gpu_memory_utilization 0.5 \
  --trust-remote-code \
  --enable-prefix-caching \
  --max-num-seqs 16 \
  --max_model_len 32768 &

See quick_vllm/vllm_args.py for a whirlwind of all server flags.

---

Python usage

1 • Functional API

from quick_vllm import send   # or `import quick_vllm.api as qvapi`

msgs = [
    "Hi there, how are you?",
    [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user",   "content": "Summarise Moby-Dick in two lines."},
    ],
]

responses = send(
    msgs,
    temperature=0.7,
    top_p=0.95,
    n=4,                 # generate 4 responses per prompt
    just_return_text=1,  # -> list[str] instead of list[dict]
    stream_print=True,   # stream-print tokens for the first prompt
)
print(responses)

Need asynchronous results? Use async_=True or the convenience wrapper send_async:

handle = send(msgs, async_=True)

# Fetch them all at once
all_responses = handle.get()

# Or grab each one as soon as it's ready
for resp in send_async(msgs):
    print(resp.get())

2 • Object-oriented client

Need to hit multiple vLLM back-ends in the same programme? Use the VLLMClient class:

from quick_vllm import VLLMClient

east = VLLMClient(host="10.0.0.11", port=8000)
west = VLLMClient(host="10.0.0.12", port=8001)

print(east.send_message("Ping east", just_return_text=True))
print(west.send_message("Ping west", just_return_text=True))

batch = ["Explain quantum tunnelling in 2 lines.", "Write a haiku about GPU fans."]
print(east.send(batch, just_return_text=True, stream_print=True))        # multiprocess batching still works

Want to use LM Studio?

from quick_vllm import VLLMClient

lms_client = quick_vllm.VLLMClient(
    host="192.168.1.169",
    port=1234,
)

example_list = [
    "Hi.",
    "How are you?",
]

scheduled_actions = lms_client.send_async(example_list)

for idx, action in enumerate(scheduled_actions):
    gotten = action.get()
    print(f"Scheduled Action {idx}: {gotten}")

Both the functional helpers and the class share the same cache directory, so you can mix and match freely.

---

Settings quick-reference

Note For VLLMClient objects just pass port / host to the constructor. The class is completely self-contained.