End-to-End Encrypted (E2EE) LLM System

This repository provides a highly secure, client-server system for hosting large language models (like HauhauCS/Qwen3.5-35B-A3B-Uncensored-HauhauCS-Aggressive).

The system guarantees that user data is protected against interception and that the remote server itself does not persist or leak user prompts or responses.

---

🏗️ Architecture

• Server: A stateless, remote E2EE server that hosts the LLM. It receives AES-encrypted payloads, decrypts them purely in-memory using its RSA private key and session AES keys, runs inference via llama-cpp-python on your GPU, and returns AES-encrypted responses.
• Client: A local proxy server running on your machine. It exposes an unencrypted, OpenAI-compatible /v1/chat/completions API to your local applications. It handles context truncation locally, encrypts your prompt, securely communicates with the remote server, and decrypts the response.

---

🚀 Step-by-Step Setup Guide

Prerequisites

Before you begin, ensure you have the following installed on both your server and client machines:

1. Python 3.10+
2. uv (A blazingly fast Python package manager):

   curl -LsSf https://astral.sh/uv/install.sh | sh
   

3. NVIDIA Drivers & CUDA Toolkit (Server only - required for GPU acceleration).

---

Step 1: Setup the Remote Server (GPU Host)

The server needs to be compiled with CUDA support so the model runs on your GPU. The server code is already configured to offload 100% of the model layers to the GPU (n_gpu_layers=-1).

1. Clone the repository and enter the server directory:

   cd server
   

2. Install dependencies with GPU (CUDA) support: By default, llama-cpp-python installs for CPU. To force it to compile for your NVIDIA GPU, run the following commands:

   # Initialize the environment
   uv sync
   
   # Force reinstall llama-cpp-python with CUDA enabled
   CMAKE_ARGS="-DGGML_CUDA=on" uv pip install llama-cpp-python --force-reinstall --no-binary llama-cpp-python --no-cache
   

3. Run the Server: The server defaults to automatically downloading and using the HauhauCS/Qwen3.5-35B-A3B-Uncensored-HauhauCS-Aggressive-Q8_0.gguf model.

   # Start the server (binds to 0.0.0.0 so external clients can reach it)
   uv run uvicorn main:app --host 0.0.0.0 --port 8000
   

   Optional Configuration:

   • If you already downloaded a model manually, set the path: export MODEL_PATH="/path/to/model.gguf"
   • If you just want to test the networking/encryption without downloading a massive model, use Mock Mode: export MOCK_MODE="True"

---

Step 2: Setup the Local Client (Your PC)

The client runs locally on your computer and acts as a bridge between your chat apps (like Jan or SillyTavern) and the encrypted server.

1. Enter the client directory:

   cd client
   

2. Install dependencies:

   uv sync
   

3. Configure and Run the Client: You must tell the client the IP address of your remote server. Replace <remote_server_ip> with your actual server IP.

   export SERVER_URL="http://<remote_server_ip>:8000"
   
   # Start the local proxy (binds to 127.0.0.1 for security)
   uv run uvicorn main:app --host 127.0.0.1 --port 8080
   

---

Step 3: Usage with Local Apps

Now that the local client proxy is running, you can configure your local LLM application (e.g., SillyTavern, Chatbox, Jan, or any OpenAI-compatible app) to use it!

• API Type: OpenAI
• API Base URL: http://127.0.0.1:8080/v1
• API Key: sk-1234 (Can be anything, it is ignored)
• Model Name: Any (the proxy ignores it and passes everything to the remote server).

Example cURL test from your local PC:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "Hello, are you running securely?"}
    ]
  }'

---

🔒 Security Guarantees

• The remote server stores no chat history (stateless).
• The prompt is encrypted via AES-GCM on the client and is only decrypted into RAM on the server.
• The AES session key is encrypted with the server's RSA public key (forwarded on startup).
• Server system logs are explicitly silenced to prevent sensitive data leaks in console outputs.