triton-tensorrt-llm

TensorRT-LLM inference on NVIDIA Triton, with Servant-based tool server.

Components

Quick Start

# Run the demo (OpenAI Proxy + Tool Server + Open WebUI)
nix run github:straylight-software/triton-tensorrt-llm

# Point to your Triton server
nix run github:straylight-software/triton-tensorrt-llm -- --triton-url http://gpu-server:8000

# Then open http://localhost:3000 for chat UI

Individual Components

# OpenAI Proxy only
nix run .#openai-proxy

# Tool Server only (code sandbox + attestation)
nix run .#tool-server

# Open WebUI only
nix run .#open-webui

# Build a TRT-LLM engine (requires GPU)
nix build .#qwen3-32b-engine --option sandbox false

# Validate an engine
nix run .#trtllm-validate -- detect-quant /path/to/model

---

Demo

The default nix run starts a full stack:

# Default: connects to localhost:8000 for Triton
nix run .

# Connect to remote Triton
nix run . -- --triton-url http://192.168.1.100:8000

# Custom model name
nix run . -- --model llama3

# Proxy only (no WebUI)
nix run . -- --proxy-only

Environment Variables

---

Tool Server

The tool-server provides a Servant-based API with automatic OpenAPI 3.0 spec generation.

Features

• Code Sandbox: Isolated workspaces for code editing and compilation
• AST Operations: Pattern search/replace via ast-grep
• Compilation: Type-check Rust, Haskell, Lean, Dhall, PureScript
• Attestation: Ed25519 identity with signed git commits
• Tools: Web search (SearXNG), URL reading (Jina)

Endpoints

Example Usage

# Start server
TOOL_SERVER_PORT=9001 nix run .#tool-server

# Get OpenAPI spec
curl http://localhost:9001/openapi.json | jq .

# Create workspace
curl -X POST http://localhost:9001/code/workspace \
  -H "Content-Type: application/json" \
  -d '{"workspace_id": "my-project"}'

# Write a file
curl -X POST http://localhost:9001/code/write \
  -H "Content-Type: application/json" \
  -d '{
    "workspace_id": "my-project",
    "path": "src/Main.hs",
    "content": "module Main where\n\nmain :: IO ()\nmain = putStrLn \"Hello\""
  }'

# Compile (type-check)
curl -X POST http://localhost:9001/code/compile \
  -H "Content-Type: application/json" \
  -d '{"workspace_id": "my-project", "path": "src/Main.hs"}'

# AST search
curl -X POST http://localhost:9001/code/ast_search \
  -H "Content-Type: application/json" \
  -d '{
    "workspace_id": "my-project",
    "pattern": "putStrLn $_",
    "language": "haskell"
  }'

# Create attestation (requires identity configured)
curl -X POST http://localhost:9001/attest \
  -H "Content-Type: application/json" \
  -d '{
    "attest_type": "task",
    "context": "Implemented feature X",
    "thought": "Considered approaches A and B, chose A for simplicity",
    "action": "Created 3 files, modified 2",
    "coeffects": {"filesystem": "readWrite", "network": "read"}
  }'

Supported Languages

The "Language Coset" for compilation:

Architecture

lib/
├── CodeSandbox.hs    # Pure domain logic (reusable)
└── Attestation.hs    # Identity & signing (reusable)

tool-server/
├── API.hs            # Servant API type definition
├── API/Types.hs      # Request/Response with OpenAPI3 schemas
└── Server.hs         # Thin handlers over lib/

---

OpenAI Proxy

The openai-proxy provides an OpenAI-compatible interface to TensorRT-LLM.

Endpoints

Configuration

---

NixOS Module

The module is self-contained — it pulls in all dependencies including:

• Triton server with TRT-LLM backend
• OpenAI proxy (Haskell/Warp)
• Tool server (Servant + OpenAPI3)
• SearXNG (native, no Docker required)

Usage

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    triton-tensorrt-llm.url = "github:straylight-software/triton-tensorrt-llm";
  };

  outputs = { self, nixpkgs, triton-tensorrt-llm, ... }: {
    nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        # Apply the overlay to get packages
        { nixpkgs.overlays = [ triton-tensorrt-llm.overlays.default ]; }
        
        # Import the NixOS module
        triton-tensorrt-llm.nixosModules.default
        
        # Configure the service
        {
          services.triton-trtllm = {
            enable = true;
            model = "qwen3";
            enginePath = /var/lib/trtllm/engines/qwen3;
            
            triton = {
              enable = true;
              httpPort = 8000;
            };
            
            openaiProxy = {
              enable = true;
              port = 9000;
            };
            
            toolServer = {
              enable = true;
              port = 9001;
            };
            
            searxng.enable = true;  # Native SearXNG, no Docker
          };
        }
      ];
    };
  };
}

Module Options

Package Overrides

You can override packages if needed:

services.triton-trtllm = {
  enable = true;
  package.openaiProxy = myCustomProxy;
  package.toolServer = myCustomToolServer;
  # ...
};

---

Development

# Enter dev shell
nix develop

# Run tool-server tests
cd tool-server && cabal test

# Run openai-proxy property tests
nix build .#openai-proxy.tests.proxy-proptest

# Check flake
nix flake check

---

Architecture

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│   AI Agent /    │────▶│   tool-server    │────▶│   Compilers     │
│   Open WebUI    │     │   (port 9001)    │     │   ast-grep      │
└─────────────────┘     └──────────────────┘     └─────────────────┘
        │                        │
        │               ┌────────┴────────┐
        │               ▼                 ▼
        │         ┌──────────┐     ┌────────────┐
        │         │ Identity │     │ Attestation│
        │         │ Ed25519  │     │ Git Repo   │
        │         └──────────┘     └────────────┘
        │
        ▼
┌──────────────────┐     ┌─────────────────┐
│   openai-proxy   │────▶│  Triton Server  │
│   (port 9000)    │     │   (port 8000)   │
└────────┬─────────┘     └─────────────────┘
         │
┌────────┼────────┐
▼        ▼        ▼
┌─────────┐ ┌──────────┐ ┌─────────┐
│ SearXNG │ │   Jina   │ │ Metrics │
│ (8888)  │ │ (remote) │ │ (CH)    │
└─────────┘ └──────────┘ └─────────┘

---

Coeffects

The tool-server tracks resource requirements (coeffects) per endpoint:

{
  "tools": {
    "code/compile": {"filesystem": "readWrite"},
    "code/ast_search": {"filesystem": "read"},
    "code/ast_replace": {"filesystem": "readWrite"},
    "attest": {"filesystem": "readWrite", "crypto": "sign"},
    "tools/search": {"network": "read"},
    "tools/read_url": {"network": "read"}
  },
  "algebra": "Coeffect semiring: (R, join, 0, meet, 1)"
}

This supports the Aleph coeffect system for provenance tracking.

---

License

MIT