GregPumpkin

A self-hosted AI chat application built in C++ with Qt, powered by llama.cpp. Runs entirely on your own hardware — no internet required, no data sent anywhere.

---

Features

• Local inference — your data never leaves your machine
• Multiple model support — Qwen, Mistral, DeepSeek R1, Phi, MiMo and more
• Per-model tabs — separate conversation history per model
• Streaming responses — smooth token-by-token output
• RAG (Retrieval Augmented Generation) — feed the AI your own documents, books, and code
• Session history — conversations saved and reloadable per model
• Settings panel — system prompt, temperature, GPU layers, model management
• Markdown rendering — code blocks, inline code, bold text
• Python model manager — download, delete, and inspect models from the app
• Unit tested — core logic covered with Google Test

---

Requirements

System

• Linux (runs only on linux systems(tested on Nobara/Fedora))
• NVIDIA GPU recommended (CPU-only works, slower)
• 8GB+ VRAM for 7B models, 16GB+ RAM

Dependencies

C++:

sudo dnf install cmake gcc-c++ boost-devel qt6-qtbase-devel gtest-devel

Python:

pip install huggingface-hub sentence-transformers chromadb PyPDF2 psutil --break-system-packages

llama.cpp (build from source):

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=OFF   # add -DGGML_CUDA=ON if you have CUDA
cmake --build build --config Release -j$(nproc)

---

Building

git clone https://github.com/yourusername/llm-bot
cd llm-bot

# Configure
cmake -B "out/build" -DGGML_CUDA=OFF

# Build
cmake --build "out/build" --config Release -j$(nproc)

---

Setup

Use the in-app model manager (Settings → Models → Download). So you can choose whatever models fits you.

The app automatically starts llama-server in the background and connects to it.

Adding Knowledge Base Documents (RAG)

1. Open Settings → Knowledge Base
2. Click Add Document
3. Select a PDF, Markdown, or code file
4. The app indexes it automatically

Supported formats: .pdf, .txt, .md, .hpp, .cpp, .h, .py

Once indexed, the AI will automatically reference relevant passages from your documents when answering questions.

---

Project Structure

llm-bot/
├── src/
│   ├── config/
│   │   ├── config_manager.hpp       # Reads/writes config.json
│   │   └── model_config.hpp         # Model registry and prompt formats
│   ├── net/
│   │   └── http_client.hpp          # Boost.Beast HTTP client
│   ├── rag/
│   │   └── rag_context.hpp          # RAG query interface
│   ├── server/
│   │   └── server_manager.hpp       # llama-server process management
│   ├── session/
│   │   └── conversation.hpp         # Message history and prompt formatting
│   ├── storage/
│   │   ├── history_store.hpp        # Session save/load
│   │   └── sessions/                # Saved conversations (gitignored)
│   ├── ui/
│   │   ├── add_model_dialog.*       # Download new models
│   │   ├── chat_bubble_delegate.*   # Custom bubble renderer
│   │   ├── main_window.*            # Main Qt window
│   │   ├── markdown_parser.hpp      # Markdown to HTML converter
│   │   ├── message_item.hpp         # Chat message data
│   │   ├── model_tab_widget.*       # Per-model tab system
│   │   ├── session_sidebar.*        # Conversation history sidebar
│   │   └── settings_dialog.*        # Settings panel
│   ├── llm_worker.*                 # Background thread for inference
│   └── main.cpp
├── scripts/
│   ├── model_manager.py             # Download, delete, list models
│   └── rag_manager.py               # Index and query documents
├── tests/
│   ├── test_conversation.cpp        # Prompt format tests
│   ├── test_markdown.cpp            # Markdown parser tests
│   ├── test_model_registry.cpp      # Model registry tests
│   └── test_history_store.cpp       # Session storage tests
├── llama.cpp/                       # Inference engine (submodule or clone)
├── config.json                      # User configuration
└── CMakeLists.txt

---

Supported Models

Adding a new model requires two steps — add an entry to scripts/model_manager.py and src/config/model_config.hpp. See the existing entries for the pattern. I will manually add models as the project grows.

GPU Layers

The -ngl flag controls how many model layers run on the GPU:

-ngl 0   → CPU only (slowest, no VRAM used)
-ngl 16  → half on GPU, half on CPU
-ngl 32  → all layers on GPU (fastest, most VRAM)

Set gpu_layers in config.json. With 8GB VRAM and a 7B model, -ngl 32 fits comfortably. For 14B models use partial offloading (e.g. -ngl 20).

---

Tech Stack

Important note

The application is in early development, there can be a lot of bugs or not safe code, my basic goal was make my idea to live, not being perfect. Polishing will come around, I am welcome to any critics and advices!