postllm

postllm is a PostgreSQL extension that makes LLM workflows native to SQL.

It provides:

• chat and completion APIs for hosted and local inference,
• embedding/chunking/retrieval helpers for vector workflows,
• runtime and governance configuration (configure, profiles, secrets, permissions, allowlists),
• request-response helpers so SQL consumers can stay in SQL.

Start in 60 seconds

1. Install and run a local extension:

cargo install cargo-pgrx --version 0.18.0 --locked
cargo pgrx init --pg17 download
cargo pgrx run pg17

2. Load the extension and run one call:

CREATE EXTENSION postllm;

SELECT postllm.configure(
  runtime => 'openai',
  model => 'gpt-4o-mini',
  base_url => 'http://127.0.0.1:11434/v1/chat/completions'
);

SELECT postllm.chat_text(ARRAY[
  postllm.system('You are concise.'),
  postllm.user('Explain MVCC in one sentence.')
]);

Demo App

Run the bundled support-triage demo to start PostgreSQL, start a local llama.cpp runtime, seed a tiny app schema, and draft one grounded response:

./scripts/demo_quickstart.sh

Use POSTLLM_DEMO_KEEP=1 ./scripts/demo_quickstart.sh if you want to inspect the demo tables afterward.

Where to read first

Choose one path based on your role:

• Trying it out quickly → getting-started
• Running the sample app → demo
• Configuring security/governance → configuration
• Choosing a runtime model → runtime
• Copy-pasting a full workflow → cookbook
• Debugging and deployment → operations
• Understanding the architecture → architecture
• Finding every function → reference
• Role-based navigation list → docs/README.md

API families at a glance

• Session/configuration: settings, capabilities, configure, runtime_discover, runtime_ready
• Profiles/governance: profile*, secret*, permission*, model_alias*
• Message builders: message, user, assistant, parts/templates, tool helpers
• Generation: chat*, complete*, streaming, structured outputs, tools, and helpers (usage, finish_reason, etc.)
• Retrieval/embeddings: chunk_*, embed*, ingest_document, rerank, hybrid_rank, rag*

For full signatures, use the grouped list in docs/reference.md.

Runtime model

postllm exposes two runtime lanes:

• openai for hosted HTTP APIs, including OpenAI-compatible Chat Completions, Responses-style generation, native Anthropic Messages, hosted embeddings, hosted reranking, Anthropic tool use, and Anthropic URL-based image inputs.
• candle for local inference paths (starter generation, local embeddings, and local reranking).

The SQL API shape is shared across runtimes; capability checks determine what arguments are valid in each lane.

Internal structure

Current code organization:

• src/lib.rs — SQL exports and extension SQL schema.
• src/api/ — API namespace modules (config, messages, inference, retrieval, ops).
• src/api/config.rs/messages.rs/inference.rs/retrieval.rs/ops.rs — SQL-facing API implementations that keep lib.rs compact.
• Internal domain modules (backend, client, guc, catalog, permissions, http_policy, operator_policy) for request validation and execution.

Notes for production use

Inference runs inside PostgreSQL backends. Before running in shared environments, make intent checks explicit:

• runtime_discover() and runtime_ready() for environment sanity.
• permission_* for role-aware controls.
• http_allowed_hosts and http_allowed_providers for outbound HTTP policy.
• model_alias* and profiles for deterministic runtime configuration.

Validation

The repository includes Docker end-to-end smoke suites and a provider-compatibility matrix:

• ./scripts/e2e_ollama.sh for the hosted OpenAI-compatible lane via a real Ollama container
• ./scripts/e2e_llama.sh for the hosted OpenAI-compatible lane via llama-server
• ./scripts/e2e_candle.sh for the local Candle lane
• ./scripts/e2e_compat.sh to run the Ollama lane, the llama.cpp lane, and targeted OpenAI/Anthropic compatibility fixtures together
• .github/workflows/ci.yml for the GitHub Actions quality gate and cross-version/platform matrix
• .github/workflows/release.yml and .github/workflows/release-drafter.yml for tagged release artifacts, GHCR publishing, and automated release notes

They are documented in docs/operations.md.

Deeper topic docs:

• Local Candle roadmap
• Cookbook recipes
• Demo app
• pgvector integration