vLLM Qualcomm Cloud AI (QAIC) Plugin

| Qualcomm Data Center AI | Documentation | User Guide | Installation Guide |

This branch is under active development — plugin rebase to vLLM v0.23.0. This is not a stable release. For production use, please switch to the main branch or release/v0.15.0.

---

Qualcomm Cloud AI 100 is an AI inference accelerator designed to deliver exceptional performance and power efficiency for Large Language Models and other AI workloads. Built on Qualcomm's advanced HexNN architecture and Neural Signal Processors (NSPs), the Cloud AI 100 provides scalable, high-throughput inference capabilities optimized for enterprise and cloud deployments.

The vLLM QAIC plugin (vllm-qaic) is a dedicated unified backend extension that enables seamless integration of Qualcomm Cloud AI 100 Accelerators with vLLM using PyTorch and Qualcomm Cloud AI Compiler.

vllm-qaic supports two inference modes:

Important: Eager and AoT inference modes will coexist in the same environment in future releases, but for now, only one mode can be used at a time. Please ensure you are following the correct setup instructions for your selected mode of inference. Follow only the steps for your chosen mode.

---

For more information about Qualcomm Cloud AI 100, check out:

• 🚀 Qualcomm Data Center AI Solutions
• 📚 Cloud AI SDK User Guide
• 🎯 Cloud AI SDK API Reference
• ✅ Validated Models and Configurations

Prerequisites

• Hardware: Qualcomm Cloud AI card(s) (Cloud AI 100, Cloud AI 080)
• OS: Linux (Ubuntu 22.04+ recommended)
• Software:
  • Python 3.12
  • Qualcomm Cloud AI SDK >= 1.22.0
  • vLLM v0.23.0

Getting Started

Please use the following recommended versions to get started quickly:

Branches

main: Primary development branch. Contributors should develop submissions based on this branch, and submit pull requests to this branch.

v0.23.0: Active development branch for plugin rebase to vLLM v0.23.0. Some features (SpD, LoRaX) are not yet ported. For production use, stay on main.

Installation

For full installation instructions covering both AOT and PYT modes, scripted and manual steps, and wheel-based installs, see the Installation Guide.

Quick start — activate a Python 3.12 environment, then:

# AOT mode
./scripts/install.sh aot

# PYT mode
./scripts/install.sh pyt

Before installing, ensure your system has the Qualcomm Cloud AI SDK and drivers installed:

1. Install Cloud AI SDK and Drivers
2. Verify your installation

PYT mode only: install the Apps SDK with --install-torch-qaic to build torch_qaic wheels into /opt/qti-aic/integrations/torch_qaic/.

---

Run an example

Set device visibility before running:

export QAIC_VISIBLE_DEVICES=0   # comma-separated device IDs, e.g. "0,1,2,3"

PYT (Eager) Mode

from vllm import LLM, SamplingParams

llm = LLM(
    model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    max_num_seqs=8,
    max_model_len=2048,
    enable_prefix_caching=False,
    gpu_memory_utilization=0.9,
    tensor_parallel_size=1,
    enforce_eager=True,
    async_scheduling=False,
)

prompts = ["Hello, my name is", "The future of AI is"]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=128)
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(f"Prompt: {output.prompt}")
    print(f"Generated: {output.outputs[0].text}")
    print("-" * 50)

AOT Mode

from vllm import LLM, SamplingParams

llm = LLM(
    model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    max_num_seqs=8,
    max_model_len=2048,
    enable_prefix_caching=False,
    tensor_parallel_size=1,
    async_scheduling=False,
)

prompts = ["Hello, my name is", "The future of AI is"]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=128)
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(f"Prompt: {output.prompt}")
    print(f"Generated: {output.outputs[0].text}")
    print("-" * 50)

Models Supported

Ahead-of-Time Compiled Mode

For the most up-to-date list of supported models and their validation status, please check Supported Models through AoT.

PyTorch Eager Mode

For the most up-to-date list of supported models and their validation status, please check Supported Models through PyTorch Eager Mode.

Support

• Documentation: Cloud AI SDK User Guide
• API Reference: Cloud AI SDK API Documentation

Development

PLease refer to CONTRIBUTING on how to submit changes.

Getting in Contact

Please report an issue or open a discussion as appropiate for your usecase.

• Report an Issue on GitHub
• Open a Discussion on GitHub

License

vllm-qaic is licensed under the Apache-2.0. See LICENSE.txt for the full license text.