liminallm

liminallm is an experiment in what a chatgpt-like system looks like if you stop hard-coding product logic and let the model help evolve itself.

the core bet: small models, deeply adapted. a small self-hosted model with behavior baked into lora weights beats a small model begging through a long system prompt - weights survive context pressure, free the window for the user's actual content, and cost nothing per token. a frontier model can help as an offline teacher, but inference never depends on one.

it’s a small kernel wrapped around:

• a frozen base llm (jax - the primary training and serving framework)
• per-user persona adapters + per-skill lora adapters trained on pooled cluster data
• the adapter ladder: skills are born as prompts and only earn weights when the data justifies it - and an eval gate agrees
• emergent “skills” from clusters + preference events
• self-describing artifacts (workflows, routing policies, tools)
• notebooklm-style grounding over filesystem-backed files
• boring infra: postgres + redis + filesystem
• artifacts and adapter payloads live as JSON/weights on the shared filesystem

the code is just the glue. everything interesting lives as data.

---

screens

the ui is one chat surface with a workspace beside it, plus a separate admin console. each image below was captured from a running instance driven through a real browser: a real server, a real postgres and redis, and a live model answering the questions shown.

sign in. email and password, with the access token held in session storage and the refresh token in an http-only cookie the page cannot read.

chat. the app rail down the left edge moves between sections; the pane beside it lists that section's items. conversations are titled from their first exchange. the thread keeps context across turns, so a follow-up that names nobody is still answered about the same subject. the bar carries the conversation's name and one action, with the knowledge context and workflow override behind its menu.

notes. a searchable vault. the pane lists and searches it, selecting a note opens it for editing, and sweep and graph operate across the whole set.

contexts. knowledge contexts group uploaded sources for retrieval. the pane lists them and the workspace shows the selected one. a conversation can be pointed at one from the chat screen's menu.

files. uploads are attached to a context, or kept private when no context is given. the same screen browses everything already uploaded.

artifacts. workflows, policies, adapters, and tools are all artifacts. the pane filters them by type and visibility; the workspace shows the selected artifact and its version history.

tools. the pane lists the registered tool specifications a workflow can call, and the workflows themselves. the workspace shows the selected one and invokes it.

insights. preference events are summarized here, so you can see what your feedback has shaped and which adapters it reached.

settings. account and session details, including the role and tenant the current session carries.

admin console. a separate page at /admin, guarded by the admin role. it reviews configuration patch proposals, administers tenant users, and inspects database objects.

---

what it does (conceptually)

feedback loop (at a glance)

User Feedback → Embeddings → Clustering → Skill Discovery
     ↑                                            ↓
Router Updates ← Eval Gate ← Adapter Training ← Prompt-Mode Skill

• chatgpt-like web ui

  • multi-user, password + pluggable auth
  • conversations, history, summaries
  • text first; voice later

• deep behavioral memory (the adapter ladder)

  • per-user persona adapters (lora): small, low-stakes - tone and format
  • skill adapters born from usage: “when problems like this show up, start with this debugging workflow”
  • every skill starts as a prompt (instructions distilled from cluster labels + highly-rated exemplars) - useful immediately on any backend
  • once a skill cluster accumulates enough independent positive evidence, it may graduate from prompt instructions to lora training. personal skills may train from one user's repeated evidence and stay private; shared skills additionally require pooled evidence from several users in the tenant. a raw count is not evidence: twenty thumbs in one long session are one episode rated repeatedly, so the gate counts distinct rated answers and distinct conversations too
  • trained weights only ship if a holdout eval gate measures real improvement; a failed gate leaves the skill on the prompt rung. nothing regresses.
  • passing the gate is the only thing that makes weights servable: the adapter's promoted version number is the authority, and serving reads exactly that version's weights. a file on disk, a latest pointer, the newest directory - none of them mean an adapter graduated.
  • a graduated skill speaks once, not twice: where its weights apply it is carried by them, and where they cannot (an api backend) its prompt carries it instead
  • optionally, a teacher model distills raw chat transcripts into clean training exemplars first
  • continuous micro-training jobs in jax, only on adapters, never on the base model

• natural factual memory

  • user files in the filesystem (/users/{id}/files)
  • ingestion → chunking → embeddings in postgres (pgvector)
  • notebooklm-style: bind “contexts” (collections of files/folders) to a chat and ask questions grounded in that corpus

• a notes vault with a witness

  • notes link to each other with [[title]]; links become a graph you can see
  • the model can search your vault mid-chat (note_search) and cite what you once wrote
  • the witness puts two dated notes side by side and asks how they relate - agrees, contradicts, or the position quietly moved. contradiction isn’t the goal; it’s one honest result of the process
  • when a position has moved, the report shows the trail: the chain of links between the two thoughts, with dates
  • a vault-wide sweep runs the same process over the strongest pairs across everything you’ve written
  • uploaded files stay chat-scoped by default; a file joins the vault only when you promote it (one click), because permanent cross-chat memory should be a decision, not a side effect
  • promoted pdfs and images get fleeced for content: text layer → pypdf, images and scans → ocr, then model vision. install tesseract-ocr + pip install 'liminallm[ocr]' - technically optional, practically required

• context that fits the model you actually run

  • the prompt budget comes from the serving model's real window - asked of the provider (gemini and vllm both report it), else a known-family table, else a conservative default; model_context_window overrides when discovery guesses wrong
  • recent turns go verbatim; older ones are folded into a rolling digest kept on the conversation, so a long chat degrades to “remembers less precisely” instead of “forgets entirely”
  • the digest is written off the hot path and never blocks a reply; the window is the same whether redis is up or down

• an openai-compatible responses api for agents

  • POST /v1/responses speaks the responses dialect, so any agent framework can point its base url here and get the kernel's whole enrichment stack - personas, skill adapters, hybrid rag, notes, memory - behind what looks like a plain model endpoint. a weak local model plus this kernel presents as a much richer model; the caller changes nothing but the base url.
  • stateful: previous_response_id continues the conversation server-side; pass context_id (a liminallm extension) on the first turn to ground the whole thread in a knowledge context
  • streaming: stream: true returns sse response.* events (created → tool items as they run → text deltas → completed), with the reply's id stable from the first event to the persisted message
  • serves what the turn learned, not just the text: server-side searches appear as file_search_call/web_search_call output items; grounding snippets, the full tool trace and active adapters ride under a namespaced liminallm key; usage includes reasoning/cached token details when the upstream reports them, and real totals from our own tokenizer on the local jax path
  • auth via api keys (sk-liminal-…): mint, list, and revoke from the settings tab in the web ui, or at /v1/auth/api-keys with a logged-in session. keys are valid only on the agent surfaces (/v1/responses, /v1/mcp) - a leaked key can chat and search and nothing else, and in particular cannot mint or revoke keys. only a sha-256 of the key is stored; the plaintext is shown exactly once, at mint time.
  • agent conversations appear in the web ui like any other chat, tagged “api” in the sidebar
  • the kernel's internal tool loop (retrieval, notes, the reranker's out-of-band scoring) rides the provider tool-call transport wherever one exists - including the local jax backend via its advertised <tool_call> channel - so agents get the same grounded answers on every backend

• an mcp server for everyone else's agents

  • POST /v1/mcp speaks the model context protocol (streamable http, revision 2025-06-18): initialize, list tools, call tools - stateless, json responses, batching rejected as the spec now requires
  • two tools, both read-only, both the kernel's own retrieval: note_search over the notes vault and knowledge_search over knowledge contexts, the exact services the internal agent loop uses
  • read-only is the point: nothing here can carry data off the box, so an injected document has no egress to abuse, and every result names its own text as document content - not instructions
  • same api keys as the responses api; the roadmap (resources, prompts, oauth, and an mcp client under the kernel's taint discipline) lives in the spec so growth is a decision, not drift

• small kernel, big data

  • kernel only knows how to:
    • auth users
    • run workflows (graphs)
    • run routing policies
    • call the llm with optional lora adapters
    • talk to postgres / redis / filesystem
  • everything else (domains, skills, behaviors, tools, routing rules) is expressed as artifacts:
    • adapter.lora
    • workflow.chat
    • policy.routing
    • tool.spec
    • context.knowledge
    • etc.

• emergent domains & skills

  • no hard-coded DEBUGGING, WRITING, whatever
  • we cluster preference events in embedding space
  • llm labels clusters (“kernel panic debugging”, “multi-tenant billing schema design”, …)
  • when a cluster is big + consistently positive, we auto-create a prompt-mode skill adapter tied to that cluster - weights come later, gated on data volume and a passing eval

• router as data, not code

  • routing policies are artifacts (policy.routing) with a tiny expression language:
    • conditions over embeddings, clusters, safety flags
    • actions: activate/deactivate adapters, scale weights, etc.
  • the router engine is dumb and stable; policy is editable data
  • a gate is an activation first and a strength second: weight 0 means the adapter is absent from the turn - no weights, no prompt, nothing sent to a provider, nothing in the kv cache key, and nothing claimed in what the turn reports it used. above zero it scales where scaling is defined; prompt text has no half-measure, so it goes in once, unchanged.

• llm as architect (under guardrails)

  • a config-ops api lets the llm propose patches to:
    • routing policies
    • workflows
    • adapter metadata
  • patches are stored, validated, can be auto- or human-approved, and are fully versioned

---

architecture (stack)

• language / runtime

  • python (services, api, orchestration)
  • jax + optax (base model, lora training, eval gates) - install with pip install -e ".[train]"
  • the local serving path is a real plain-jax decoder - rmsnorm, rope, grouped-query attention with a kv cache, swiglu - loading config.json + *.safetensors straight from the model directory (no torch, no flax). incremental decode is tested to reproduce a full recompute, and a lora adapter at B=0 is tested to change nothing. with no checkpoint on disk it falls back to a synthetic stand-in and says so in the log - that path moves tokens, it does not answer questions. training uses that same forward pass, so an adapter is fitted to the model that will serve it: the loss is taken over the real decoder with the lora matrices inside its attention projections, and weights only load onto the base they declare.
  • conversations reuse their own kv prefix across turns (content-addressed, adapter-keyed, strict-prefix only), so the reused prefill shows up honestly as cached_tokens in usage
  • remote multi-lora servers (lorax / vllm-style, openai-compatible) as an optional scale-out serving path; same artifacts, config change only

• storage

  • postgres
    • users, auth, conversations, messages
    • artifacts & versions
    • semantic clusters
    • knowledge chunks (with pgvector)
    • preference events, training jobs, router state
  • redis
    • sessions
    • rate limiting
    • hot conversation summaries
    • router and workflow scratch state
  • filesystem
    • /shared/models – frozen base model weights
    • /users/{id}/files – user docs
    • /users/{id}/adapters – per-user lora weight files
    • /users/{id}/artifacts – generated notebooks, exports, etc.

• services (logically)

  • auth service
  • chat orchestrator
  • artifact service
  • workflow engine
  • router service
  • llm inference (jax + lora)
  • knowledge / rag service
  • preference + training service
  • clusterer + skill discovery
  • configops (patch proposals / approvals)

for v1 these can all live in one python app with clear module boundaries.

---

current status

• early design / prototyping
  • do not treat as production-ready
  • interfaces & schemas are expected to change
• the training loop is real: jax + optax lora training with causal-lm sft batches, holdout eval, and a promotion gate - a skipped or regressed run never ships weights
• skill adapters follow the ladder end-to-end: prompt-mode birth → pooled-data training job → eval-gated graduation to hybrid
• goal is to keep:
  • implementation minimal
  • all "product behavior" in data (artifacts / policies / workflows)
  • evolution driven by usage + llm suggestions, not constant code surgery

---

quick start

See INSTALL.md - Docker on Linux, Linux without Docker, or OpenBSD.

acceptance criteria (ready to test)

Before QA begins, verify:

Run the automated smoke test:

./scripts/smoke_test.sh http://localhost:8000

---

deployment

• Installation in INSTALL.md; operations and backend lanes in docs/DEPLOYMENT.md
• Configuration architecture documented in docs/CONFIGURATION.md
• Testing guide in TESTING.md

implementation completeness (prototype)

• implemented
  • file upload endpoint writing to the shared filesystem and ingesting chunks into RAG contexts with configurable chunk sizes; default retrieval runs against pgvector with shared deterministic embeddings (optional in-process hybrid fallback for dev/test)
  • workflow execution with branching/parallel scheduling across workflow.chat graphs
  • router policies with a sandboxed evaluation engine (limited adapter gating usage)
  • pluggable model backend that can target external API fine-tune IDs or local JAX+LoRA adapter application
  • filesystem-backed LoRA adapter training that turns preference events into new adapter versions
  • preference capture with clustering + skill adapter promotion and routing integration
  • hardened auth + multi-tenant isolation: OAuth provider mapping, session revocation on password resets, error envelopes with stable error.code, ownership-enforced artifact and conversation access (including workflows/tools), adapter checksum + path validation, and email verification flows
  • MFA with TOTP enrollment (otpauth URL), session gating, and login verification
  • email verification tokens with /v1/auth/request_email_verification and /v1/auth/verify_email
  • tenant-scoped conversation history enforcement in workflows and tool invocations
  • HMAC-signed JWT access tokens with refresh rotation, tenant-aware sessions, and admin-only config endpoints
  • preference UI and rich routing feedback loop
  • LLM-as-architect auto-patch generation
  • voice interface
  • admin UI for patch approval
  • chat and admin frontends prompt for MFA codes when required and revoke sessions on logout

---

getting started (high level)

note: this is intentionally vague; exact commands depend on how you wire the codebase.

1. bring your infra
   • postgres (with pgvector installed)
   • redis
   • filesystem path accessible to the app

• gpu / tpu for jax model if you expect to train adapters
• backend selection is single-sourced from the SQL deployment config and editable from the admin console; env vars only override if you set them explicitly
• set model_backend to local_gpu_lora in the admin console to target the local JAX+LoRA path instead of external API fine-tune IDs; leave the default to use the OpenAI-style plug. The JAX backend (LocalJaxLoRABackend in liminallm/service/model_backend.py) loads adapters from the filesystem, tokenizes prompts, runs a JAX forward pass, and enforces conservative shapes; it requires a JAX runtime and optionally a Transformers tokenizer for decode parity. Provider keys are admin settings, with <PROVIDER>_API_KEY as an environment fallback.

frontend (chat + admin)

• The chat UI lives in /frontend and is served by the FastAPI app at /, with static assets mounted at /static/*. It is a three-band shell: a 48px rail for sections, a contextual pane listing that section's items, and the workspace beside them.
• Authenticate with /v1/auth/login. The access token is held in session storage and sent as a bearer token; the refresh token and session id are http-only cookies the page cannot read.
• The admin console is separate at /admin and is guarded by the admin role (FastAPI enforces the role before serving the HTML). It surfaces config patch proposal/approval flows backed by /v1/config/* endpoints, tenant-scoped user administration (list/add/delete, role changes), adapter visibility, and a read-only inspector for database objects.

adapters: local LoRA vs remote fine-tune IDs vs prompt-distilled

• Router policies pick an adapter; the inference backend decides whether that means applying LoRA weights locally, swapping to a remote fine-tuned model ID, or injecting distilled prompt instructions on top of a black-box API.

• Each adapter.lora artifact carries a backend field describing where inference happens:

  {
    "kind": "adapter.lora.remote",
    "provider": "zhipu",
    "backend": "api",
    "base_model": "glm-4-air",
    "remote_model_id": "glm-4-air-ft-2025-11-01-u123-debug",
    "region": "cn-beijing",
    "cluster_id": "…",
    "applicability": {
      "natural_language": "u123: kernel panic debugging skill on GLM-4-Air",
      "embedding_centroid": []
    }
  }
  

  {
    "kind": "adapter.lora.local",
    "backend": "local",
    "provider": "aliyun",
    "base_model": "qwen2.5-32b-instruct",
    "cephfs_dir": "/users/u123/adapters/{id}",
    "rank": 8,
    "layers": [0, 1, 2, 3],
    "matrices": ["attn_q", "attn_v"],
    "cluster_id": "…"
  }
  

  {
    "kind": "adapter.lora.prompt",
    "backend": "prompt",
    "provider": "api_only",
    "base_model": "glm-4-air",
    "prompt_instructions": "for kernel issues: reproduce → bisect → log inspection; keep replies terse",
    "cluster_id": "…",
    "applicability": {
      "natural_language": "prompt-distilled skill for kernel debugging",
      "embedding_centroid": []
    }
  }
  

• Remote adapters send requests to OpenAI-compatible fine-tuned model IDs (e.g., Zhipu BigModel or Alibaba DashScope). Local adapters resolve to filesystem-backed LoRA weights and are composable. Prompt-distilled adapters inject behavior as system messages without changing model IDs so you can still steer API-only providers.

• “Model-ID adapters” (fine-tuned endpoints) map 1:1 to model strings on providers like OpenAI/Azure (fine-tuned deployments), Vertex AI Gemini, or Bedrock custom models. Switching behavior = switching the model string; composition happens at routing time, not inside a single call.

• “Adapter-ID adapters” (multi-LoRA / adapter servers) surface adapter_id parameters on Together AI Serverless Multi-LoRA, LoRAX-style servers, or SageMaker adapter inference components. The backend keeps the base model string and passes adapter_id for one-or-more adapters per request when supported.

• Hybrid patterns (local adapter-enabled “controller” + external API “executor”) flow through the same artifacts: the controller uses a local LoRA backend to plan, then the API backend executes with prompt or remote-model adapters.

2. configure env - one variable

   • DATABASE_URL – postgres dsn. that is the configuration.

   four others exist and none of them are settings you tune: BUILD_SHA (stamped by the build), TEST_MODE (the test harness), EMBEDDING_VECTOR_DIM (a property of the schema you applied, shared with scripts/migrate.sh), and EXTRACT_READER_PLUGINS (imports python modules, so making it settable from a web form would mean remote code execution).

   everything else - the model, credentials, rate limits, ttls, cors, smtp, the signing key - lives in the database and is edited from the admin console at /admin, applied to every replica without a restart. changing an smtp password should not require redeploying the app. for declarative deploys, seed on first boot with INSTANCE_SETTINGS_JSON='{"model_backend": "stub"}'.

3. migrate db

   • run the alembic / migration tool to create tables described in the spec.
   • if you ran earlier builds, delete ${SHARED_FS_ROOT}/state/training_pg.json after upgrading to purge legacy MFA secrets (secrets are now sourced solely from the user_mfa_secret table).

4a. preference_event → adapter dataset → tokenized batches

• preference_event rows (positive feedback) capture context_embedding, score, and optional context_text; they are clustered per-user to build adapter personas.
• the training service reconstructs prompts from recent messages, appends any provided context snippet, and uses corrected text as targets while tracking cluster centroids.
• dataset rows are written to ${SHARED_FS_ROOT}/users/{user_id}/adapters/{adapter_id}/jobs/{job_id}/dataset.jsonl.
• tokenized batches carry shapes for the downstream JAX/Optax loop (padding + masks, no base-model update), and training metadata records batch shapes + cluster summaries.
• adapter metadata and params are stored under ${SHARED_FS_ROOT}/users/{user_id}/adapters/{adapter_id}/v####/.

4. start services

   • run the api server (http + websocket for streaming)
   • run a background worker for:
     • ingestion / embeddings
     • clustering
     • adapter training
     • configops patch application

5. open the web ui

   • sign up / log in
   • create a conversation
   • upload a few files, create a knowledge context, and attach it to a chat
   • start talking to see basic chat + rag behavior
   • enable preference capture + adapters once that’s wired

---

roadmap (rough)

• minimal chat with postgres-backed conversations
• file upload + filesystem + rag over pgvector chunks
• artifacts for workflows + tools (no adapters yet)
• preference events + single persona adapter per user
• semantic clustering + skill adapters
• router policies as data + simple editor
• configops api + llm-generated patches
• mobile / voice clients (optional layer)

---

license

MIT

testing

TESTING.md documents the lanes in full. The common ones:

make test-fast-xdist   # the default edit-loop lane, about two minutes
make test-xdist        # every test but the browser lane, in parallel
make test-browser      # the browser lane; needs a Chromium binary

The suite starts its own throwaway Postgres and Redis (tests/harness.py) and applies sql/schema.sql, so tests exercise the same store and cache the app runs against. Set TEST_DATABASE_URL / TEST_REDIS_URL to point at existing services instead.

CI runs the same selection on Python 3.10, 3.11 and 3.12. The wall-clock performance tests run in their own serial pass, so four parallel workers are not competing with the thing being timed.

For a full integration run against containers:

docker compose -f docker-compose.test.yml up --build
./scripts/smoke_test.sh

API endpoints

Key endpoints (Bearer access token required):

• POST /v1/auth/signup → returns session + signed access/refresh tokens
• POST /v1/auth/login → returns tokens, with MFA gating when enabled
• POST /v1/auth/refresh → rotates refresh tokens
• POST /v1/chat → creates conversation + LLM reply
• POST /v1/responses → the same turn in OpenAI's Responses shape, for agents (api key or session auth; stream: true for SSE)
• POST /v1/mcp → MCP server (note_search + knowledge_search) for MCP-speaking agents
• POST /v1/auth/api-keys → mint an agent-surface api key (list with GET, revoke with DELETE /v1/auth/api-keys/{id})
• GET /v1/artifacts → lists data-driven workflows/policies
• GET /v1/admin/settings → admin-only system settings

Admin endpoints (/v1/admin/*, /v1/config/*) require admin role.

operational hardening

• local rate limits fall back to in-process counters when Redis is unavailable (TEST_MODE), covering auth and chat flows
• uploads are capped by max_upload_bytes to prevent unbounded in-memory reads