🐝 Alveare

An open-source LLM inference runtime and compiler for the AMD Ryzen™ AI (XDNA2) NPU on Linux.

Alveare runs large language models on the AMD Ryzen AI NPU with a 100% open-source stack — including open AIE hardware kernels (MLIR-AIE / IRON), a high-throughput native C++ inference engine (alveare_runtime), a Python control server, and an interactive React Web UI dashboard.

Alveare 2.0: Supports THREE Gemma model families end-to-end on the NPU (Gemma-4-12B, Gemma-4-E4B, Gemma-3-1B) and Llama-3.2-1B. Includes a full-featured React Web UI Dashboard with in-browser Hugging Face model setup, real-time streaming chat, conversation history management, multimodal file upload, generation sampling controls (temperature, top_p, top_k, enable_thinking), and a dark/light theme switcher.

---

🌟 Key Features

• 🚀 Native C++ Inference Server (alveare_runtime): Zero Python overhead in the execution loop. Direct XRT device management, fused FFN NPU kernels (gate/up + GeGLU + down), native KV caching, and multi-core GEMV tiled across all 32 NPU compute tiles.
• 🖥️ Modern React Web UI Dashboard: A sleek, real-time control center at http://127.0.0.1:8000. Monitors NPU telemetry, streams server logs live, manages model lifecycle, and provides an interactive chat playground.
• ⚡ In-Browser & Automated 1-Click Model Setup: "Aggiungi Modello" modal enables 1-click GGUF downloading and quantization from Hugging Face for Gemma-4 12B, Gemma-4 E4B, Gemma-3 1B, and Llama 3.2 1B.
• 💬 Rich Chat Experience: Real-time token streaming with Markdown & code highlighting, persistent multi-turn conversation sessions (sidebar history manager), multimodal file upload (PDF/text/image/audio parsing), and dark/light theme toggle.
• 🎛️ Advanced Sampling Controls: Full control over temperature (0.0 to 2.0 with greedy decoding at 0.0), top_p (nucleus sampling), top_k, max_tokens, max_context_length, system prompt customization, and enable_thinking toggle for reasoning models.
• 🎙️ Real-Time Speech-to-Text (Whisper Base & Whisper Large v3): Integrated speech recognition. Supports live microphone voice recording in Web UI, automatic audio file transcription (MP3/WAV/M4A/OGG/FLAC/WebM), and OpenAI-compatible /v1/audio/transcriptions API with accurate multilingual transcription (Italian, English, European & Asian languages) on AMD NPU and CPU.
• 🗣️ Neural Text-to-Speech & Voice Cloning (Audio8 TTS 0.1B & 0.6B): High-fidelity neural voice synthesis with natural Italian prosody, zero-shot voice cloning, and OpenAI-compatible /v1/audio/speech API accelerated on AMD NPU and CPU.
• 🔌 OpenAI-Compatible API: Standard /v1/chat/completions, /v1/audio/transcriptions, /v1/audio/speech, and /v1/models REST endpoints for direct drop-in replacement with the OpenAI Python SDK, LangChain, LlamaIndex, Continue, etc.
• 🧩 Custom Quantizer Plugin Architecture: Bring any custom architecture to the NPU by implementing BaseQuantizer (tools/convert/base_quantizer.py).

---

📦 Supported Models & NPU Performance

⚡ Performance Transparency: All models execute natively across all 32 AIE2 cores with zero-bubble asynchronous pipelining, saturating the physical memory bandwidth of the AMD Ryzen AI 300 series processor.

---

🚀 Quick Start

1. One-Command Installation

./install.sh

2. Verify NPU Hardware

./alveare check

3. Launch Web UI & Server

./alveare start

Open http://127.0.0.1:8000 in your browser.

4. Pick or Add a Model

Click "Aggiungi Modello" in the Web UI to download and quantize Gemma 4 12B, Gemma 4 E4B, or Gemma 3 1B directly from Hugging Face in one click.

Or from the command line:

./alveare setup gemma4 --arch gemma4

For the complete guide, see 🚀 Quickstart Guide.

---

📚 Documentation & Guides

• 🚀 Quickstart Guide: Step-by-step 5-minute setup and usage.
• 📦 Adding & Quantizing Models: 1-click HF downloads, CLI setup, manual GGUF quantization, and quantizer plugins.
• 🎛️ Sampling Controls & API Reference: Sampling parameters (temperature, top_p, top_k, seed, enable_thinking, system prompt) and OpenAI API schema.
• 🎙️ Speech-to-Text Engine Guide: Whisper STT on AMD NPU, WebSocket streaming, and REST API.
• 🗣️ Text-to-Speech Engine Guide: Audio8 TTS on AMD NPU, voice cloning, and OpenAI speech endpoint.
• 🛠️ Full Hardware & Toolchain Setup: Deep dive into Linux driver permissions, XRT libraries, and AIE compiler requirements.
• 🔌 Custom Quantizer Plugins: Build custom tensor tiling and layout plugins for new model architectures.
• 🏛️ System Architecture: Overview of C++ engine, Python control server, and AIE kernel pipeline.

---

🛠️ The alveare CLI Launcher

alveare install                               One-command full setup (env + toolchain + build).
alveare check                                 NPU smoke test and driver verification.
alveare quantize [alias] <gguf> [--arch A]    Quantize local GGUF into Alveare Q4 weights.
alveare setup <alias> --arch A [--url URL]    Download HF GGUF and quantize in one step.
alveare list  (or: models)                    List all installed model packages.
alveare delete <model_id>                     Remove a quantized model package from disk.
alveare serve [model] [--host H] [--port P]   Launch C++ runtime & Python API/Web UI server.
alveare chat [--host H] [--port P]            Interactive terminal chat client.
alveare help                                  Show help message.

---

⚡ OpenAI API Usage Examples

Alveare's API server is a drop-in replacement for OpenAI endpoints.

Python SDK

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="gemma4",
    messages=[
        {"role": "system", "content": "Be direct and helpful."},
        {"role": "user", "content": "Explain NPU acceleration in three bullet points."}
    ],
    temperature=0.7,
    top_p=0.9,
    top_k=40,
    max_tokens=128,
    stream=True
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="", flush=True)
print()

Streaming via curl

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma4-e4b",
    "messages": [{"role": "user", "content": "Ciao! Chi sei?"}],
    "temperature": 0.7,
    "top_p": 0.9,
    "stream": true
  }'

---

🔬 Hardware & Reference Environment

Detailed hardware architecture & compiler notes are in docs/hardware.md and docs/background.md.

---

📁 Repository Structure

alveare               Top-level CLI launcher script
frontend/             React Web UI dashboard (Vite + Tailwind/CSS)
runtime/cpp/          Native C++ inference server (alveare_runtime) & XRT engine
runtime/py/           Python control server (control_server.py) & HTTP REST API
kernels/              Open AIE hardware kernel sources (IRON / MLIR-AIE)
tools/                Model quantizers, setup scripts, and conversion plugins
docs/                 Architecture documentation, setup guides, and specifications
tests/                Correctness test suites & microbenchmarks

---

📄 License

MIT License — See LICENSE for details. Alveare is 100% open-source, including all AIE hardware kernels.