DerridAI

English · Français · Español · Português · Deutsch · Italiano · Magyar · Русский · हिन्दी · বাংলা · العربية · 简体中文

DerridAI is a local-first, provenance-preserving research environment for building, reviewing, searching, and querying scholarly corpora. It combines source ingestion, human-in-the-loop corpus construction, evidence-aware metadata enrichment, derived vector/search indexes, and evidence-grounded retrieval-augmented generation (RAG) in one Docker application.

DerridAI is also the originating reference implementation of cELF 1.0 — the Capta-Enriched Locatable Factum, a provenance-preserving information architecture for AI-assisted documentary research. The implementation keeps source identity, record identity and revision, metadata assertions, evidence, generated claims, and support bindings separately inspectable instead of flattening them into one opaque vector store.

Current version: 0.82.0 — Gloucester (release notes). This README describes the Gloucester release.

What DerridAI does

• Acquire and ingest heterogeneous sources. Upload PDF, plain text, RTF, DOCX, images, and audio; import URLs and Project Gutenberg material; or use Corpus Capture to discover and acquire works through provider adapters such as Wikidata, Project Gutenberg, and Wikisource. Ingestion applies medium-specific safety/resource limits and preserves extractor/tool/version provenance.
• Build corpora with media-aware workflows. Corpus Builder separates source registration, extraction/transcription, source-unit mapping, structure/segmentation, enrichment, record construction, review, and publication. Controls and evidence coordinates adapt to the source medium instead of applying PDF/page concepts to everything.
• Preserve cELF provenance and field authority. Canonical FieldAssertion records distinguish derivation, evaluation, authority, value state, confidence, evidence, actor/model, stable field identity, and record revision. Human confirmation does not erase model or deterministic provenance.
• Review records with evidence in context. Reviewers can edit text and metadata, inspect source spans and other-record evidence, compare revisions, use semantic maps and relationship views, accept or reject suggestions, and publish with auditable decisions. Optimistic review saves keep the interface responsive while conflicting same-record writes are serialized.
• Use configurable metadata schemas and reviewed precedent. Schemas define stable fields, types, controlled values, evidence/review rules, POS/NER hints, field scope, model guidance, and retrieval policy. Reviewed examples become bounded, evidence-linked precedents for later enrichment without replacing canonical reviewer decisions.
• Add optional Document Intelligence. A provider-neutral derived analysis layer can add entities, coreference, quotation-speaker information, and semantic-content relationships. Small English and French spaCy models plus LatinCy for academic/classical Latin are bundled; other digest-verified language packs are administrator-managed, and an isolated BookNLP worker remains an optional English enhancement. These annotations remain rebuildable analysis, not source evidence or corpus authority.
• Search derived projections without confusing them with the corpus. ChromaDB stores rebuildable semantic/search projections and caches, using embedded or HTTP-server modes. Dense, lexical, MMR, reciprocal-rank fusion, filters, language routing, and bounded cross-encoder reranking are available where appropriate.
• Run evidence-grounded Research/RAG. Research supports hybrid retrieval, reranking, selected-evidence mode, evidence budgets, optional Record-size-aware retrieval sizing, streamed/cancellable generation, deterministic citation rendering, claim/support persistence, claim validation, response/claim memory, and LLM grading. Provenance-incomplete records are excluded from evidence rather than silently treated as valid support.
• Publish portable research sites. Works can export immutable static snapshots with Works/Record browsing, metadata filters, local annotations, keyword/semantic/hybrid search, browser-ready vectors or local Transformers.js indexing, reader-configured generation providers, and evidence-grounded Research. Generated answers bind deterministic inline citations back to the corresponding evidence, and the published interface has its own Vue entrypoint over the framework-neutral DerridAI SDK.
• Inspect and configure AI pipelines. Pipeline Studio separates server-owned workflow purpose, strategy family, and scholarly effect; exposes typed stage inputs/outputs, resolved wiring, versioned definitions and assignments, traces, scope/latency/complexity metrics, retrieval tuning ports, non-persistent comparison, and fixed-case benchmarks. Retrieval, memory, metadata-precedent, reviewer-evidence, recovery, segmentation, and enrichment stages can be explicit while provenance/authority gates remain structural constraints.
• Expose distinct API transports for distinct jobs. REST owns commands and mutations; a read-only cELF-aware GraphQL façade composes typed reads; an authenticated WebSocket plane sends realtime operation notifications. Realtime messages are never canonical state and clients can resynchronize from REST/GraphQL.
• Support controlled multi-user research. Built-in Administrator and Researcher roles plus custom researcher-safe roles are enforced by both UI and API. Researcher-facing source text is summarized at the API boundary, jobs are owner-scoped, and administrator-only corpus/system mutations remain inaccessible.
• Keep long-running work observable and pages usable while data changes. Corpus builds, LLM review, RAG, grading, imports, model/language-pack work, and vector upserts surface as cancellable operations with durable snapshots/history and realtime progress. Resource-change notifications trigger canonical refetches without making WebSocket state authoritative, and progressively loaded workspaces retain usable content, isolate failures, and reject stale responses.
• Ship an accessible, multilingual interface. English and Canadian French are first-class UI locales with enforced key parity; README coverage is broader. WCAG 2.2 AA, keyboard access, visible focus, reflow, reduced motion, forced-colors/high-contrast behavior, and long-string/localization checks are release criteria. The Help Center is search-first, role-aware, bookmarkable, and includes task shortcuts, route guidance, workflow FAQs, and a technical/plain-language glossary.
• Back up the research environment. Backup/restore covers workspaces, audit history, provider profiles, source assets, system/provenance state, and Chroma collections/embeddings.

Current master

The summary below describes the Gloucester release.

• Corpus Builder is now a progressive Setup → Build → Review → Publish workflow. It supports bounded concurrent enrichment, explicit corpus topology and Record-sizing choices, resumable/revision-aware review, persistent record-local review queues, focused publication-remediation sessions, repeatable metadata groups, audio speaker assignments, and safe text review while verified preparation is still in progress.
• Metadata and evidence processing does less model work while preserving stronger semantics. Deterministic/candidate-first routing, semantic identity and value equivalence, support-validated evidence cascade v2, structured-completion repair/retry classification, and incremental Metadata Memory/exemplar reconciliation reduce latency without allowing retrieval relevance or malformed output to become evidence.
• Pipeline Studio models executable computation explicitly. Server-owned purposes, strategy families, scholarly effects, typed ports, resolved wiring, stage traces, scope/complexity metrics, tunable retrieval parameters, and non-persistent comparison now cover more Search, Research, reviewer-evidence, recovery, segmentation, and enrichment paths.
• Works can publish portable research sites. Static exports combine the DerridAI SDK with a dedicated Vue runtime for browsing, annotations, browser semantic indexing, reader-configured providers, and citation-bound Research without turning the export into canonical corpus state.
• Realtime invalidation and progressive loading replace broad polling and blank-page refreshes across more of the application. Works, Record, Search, Research, Response Library, Languages, Relationships, Accounts/Roles, Metadata Memory, and related surfaces retain useful content during refreshes, expose local retry states, and discard stale responses.
• The frontend is continuing to retire the legacy runtime. Router-owned navigation, Vue-hosted dialogs and notifications, shared domain/state modules, Pinia slices, and extracted provider/search/compare/workspace helpers reduce coupling while the remaining compatibility code stays isolated.
• cELF and developer infrastructure were tightened. The specification is product-neutral and profile/provenance-oriented with generalized EvidenceRef locator semantics; architecture maps now document critical code boundaries; CI/pre-push selection, repository hygiene, and copyright enforcement were strengthened.

See the User Guide for the full feature reference.

cELF traceability model

DerridAI's scholarly data model follows the cELF distinction between authoritative documentary/scholarly state and rebuildable computational projections. At full traceability, the conceptual path is:

SourceDocument
  -> SourceSpan
  -> Record
  -> RecordRevision
  -> FieldAssertion
  -> Evidence Acquisition
  -> EvidenceRef
  -> EvidencePacket
  -> GenerationRun
  -> GeneratedClaim
  -> SupportBinding

This means an answer can be audited backward from a generated claim to its support, evidence, record revision, source span, and source document. Embeddings, retrieval rank, reranker scores, caches, UI state, and other operation-specific values remain derived state; they do not become intrinsic properties of the source record.

See SPECIFICATION.md for the normative cELF 1.0 specification.

Architecture

DerridAI's top-level dependency and authority boundaries are:

flowchart LR
    Browser["web · Vue 3 browser app"]

    subgraph API["api · FastAPI"]
        REST["REST commands + mutations"]
        GQL["GraphQL read façade"]
        WS["WebSocket notifications"]
        Reads["cELF read services"]
        Domains["Corpus · Research · Pipelines · Jobs"]
    end

    Canonical["Canonical scholarly state<br/>sources · records · revisions · assertions · evidence"]
    System["Server-owned durable state<br/>SQLite"]
    Chroma["Derived/rebuildable projections<br/>Chroma + embeddings + caches"]
    Providers["LLM / embedding providers"]
    NLP["Optional document-nlp worker"]

    Browser --> REST
    Browser --> GQL
    Browser <-->|bounded events| WS
    REST --> Domains
    REST --> Reads
    GQL --> Reads
    Reads --> Canonical
    Reads -. query derived projections .-> Chroma
    Domains --> Canonical
    Domains --> System
    Domains -. index / retrieve .-> Chroma
    Domains --> Providers
    Domains -. change notifications .-> WS
    Domains -. bounded reviewed text .-> NLP
    NLP -. derived annotations .-> Domains

For developers working inside a subsystem, the local maps are more specific than this overview:

• Backend application map
• Backend test architecture
• Computational pipelines
• Frontend workspace
• Frontend application map
• Frontend domain layer
• Frontend components
• Frontend API clients
• Corpus Builder frontend feature
• Corpus Builder components
• Pipeline Studio components
• Frontend test architecture

Contributing by area

Read docs/CODE_READABILITY.md before a refactor. It records naming, documentation, nesting, and provenance-preservation conventions, plus the main traps in legacy/derived-state code.

Runtime services

• web — Vue 3, TypeScript, Pinia, Vue Router, Vite, PDF.js, and nginx. It is the browser application, proxies /api/, and consumes REST, GraphQL, and realtime notifications. Storybook is an opt-in development profile.
• api — Python 3.12, FastAPI, Strawberry GraphQL, the ChromaDB client, PyMuPDF, sentence-transformers, and spaCy. It is the authoritative application boundary for authentication, source/corpus operations, cELF reads, provenance, RAG, pipelines, jobs, and system state.
• document-nlp — an optional isolated BookNLP worker for English Document Intelligence. It receives bounded reviewed text and has no corpus authority.
• chroma — an optional HTTP Chroma server. Embedded PersistentClient remains the default; both modes store derived search/vector projections.
• ollama — an optional local Ollama service. DerridAI can instead use Ollama already running on the host or any configured OpenAI-compatible endpoint.

The default Compose stack starts web and api; the other services are opt-in profiles or external providers.

Authority and persistence

DerridAI deliberately does not treat every store as equally authoritative:

• Canonical scholarly state — source assets/identity, records and revisions, field assertions, review decisions, exact evidence/support bindings, and publication state.
• Server-owned durable state — authentication plus system/provenance/job/pipeline state in SQLite under ./data.
• Derived/rebuildable state — Chroma indexes, embeddings, metadata-exemplar projections, retrieval scores, semantic-content projections, Document Intelligence output, and caches.
• Browser workspace state — local working-set preferences and unsaved workspace state, kept separate from corpus authority.

Transport split

• REST: every command and mutation, including uploads, review decisions, jobs, publication, administration, backup, and restore.
• GraphQL: read-only cELF-aware typed query façade at POST /api/graphql; no mutation or subscription root.
• WebSocket: authenticated realtime notification plane at WS /api/ws/events; never a source of truth.

For detailed module ownership, persistence boundaries, and data flow, see Architecture, GraphQL, and Realtime.

Getting started

These are the supported clean-checkout steps.

1. Prerequisites

Install Git, Docker Engine/Desktop with the docker compose command, and an LLM endpoint. The default configuration expects Ollama on the host.

Default models:

gemma4:e2b
bge-m3:latest

2. Clone and configure

git clone https://github.com/ajschlosser/DerridAI.git
cd DerridAI
cp .env.example .env

PowerShell:

Copy-Item .env.example .env

On Docker Desktop with WSL, set HOST_UID and HOST_GID in .env to the output of id -u and id -g so bind-mounted SQLite/Chroma files remain owned by your host user.

Do not export DerridAI test-storage variables such as CHROMA_DATA_ROOT, AUTH_DB_PATH, SYSTEM_DB_PATH, or CHROMA_PATH globally in your shell. Compose interpolates exported variables before values from the file are passed into the container.

3. Make the configured models available

For Ollama already running on the host:

ollama pull gemma4:e2b
ollama pull bge-m3:latest

The default .env.example uses:

OLLAMA_BASE_URL=http://host.docker.internal:11434
OLLAMA_MODEL=gemma4:e2b
OLLAMA_EMBED_MODEL=bge-m3:latest
EMBEDDING_PROVIDER=ollama

Or use the optional Compose Ollama service:

docker compose --profile ollama up -d ollama
docker compose exec ollama ollama pull gemma4:e2b
docker compose exec ollama ollama pull bge-m3:latest

Then set OLLAMA_BASE_URL=http://ollama:11434 in .env.

4. Start DerridAI

docker compose config --quiet
docker compose up -d --build

Default endpoints:

• Application: http://localhost:8181
• API: http://127.0.0.1:8000
• OpenAPI docs: http://127.0.0.1:8000/docs

On first launch, create the initial administrator account in the browser. DerridAI ships no default credentials.

5. Verify the installation

docker compose ps
curl -fsS http://127.0.0.1:8000/api/live

The liveness response should contain "ok": true, the application version, and the baked git commit when available.

For a broader diagnostic:

./scripts/diagnose.sh

PowerShell:

.\scripts\diagnose.ps1

6. Optional services

# Local Ollama service
docker compose --profile ollama up -d ollama

# Chroma HTTP server (then set CHROMA_MODE=http)
docker compose --profile chroma up -d chroma

# English BookNLP Document Intelligence enhancement
docker compose --profile document-nlp up -d document-nlp

# Storybook development surface
docker compose --profile dev up storybook

See Document Intelligence before enabling or installing NLP language packs.

7. Stop or rebuild

docker compose down

# after pulling changes
docker compose down
docker compose up -d --build

If an older release left root-owned files under data/, run ./scripts/fix-data-permissions.sh on supported Unix-like hosts.

Developer setup

CI uses Python 3.12 and Node 22.

Backend/test environment:

python3.12 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
pip install -r api/requirements-dev.txt

Frontend environment:

cd web
npm ci --no-audit --no-fund
npx playwright install chromium
cd ..

Fast local quality gates:

ruff check api/app tests scripts/check_frontend_api_contract.py scripts/check_frontend_graphql_contract.py
mypy
python -m compileall -q api/app
pytest -q -n auto --dist=worksteal \
  --ignore=tests/test_frontend_api_contract.py \
  --ignore=tests/test_frontend_graphql_contract.py
pytest -q -m contract tests/test_frontend_api_contract.py tests/test_frontend_graphql_contract.py

cd web
npm run format:repo:check
npm run lint
npm run typecheck
npm run typecheck:tests
npm run test:unit
npm run build:ci

Use npm run format:repo from web/ to format every Prettier-supported source, configuration, and documentation file in the repository. Generated legacy DOM snapshot HTML is intentionally excluded.

For a change-aware local gate, run scripts/preflight.sh from the repository root after installing both environments. It selects checks from the diff and verifies that the branch is not behind origin/master; browser suites remain CI-owned.

For browser coverage, Storybook, CI parity, and contribution rules, see CONTRIBUTING.md.

Choose a contribution area

Start with the narrowest architectural map for the part you intend to change. This avoids learning the whole repository before making a focused contribution.

Repository map

• api/app/ — FastAPI backend: source/corpus workflows, cELF read services, GraphQL, realtime, provenance, pipelines, RAG, providers, persistence, and jobs.
• web/src/ — Vue 3 application: views, components, Pinia stores, routing, API clients, realtime client, domain modules, and the shrinking isolated legacy compatibility layer.
• web/sdk/ — framework-neutral TypeScript SDK for publication loading, local retrieval, citations, annotations, provider injection, and Research.
• web/site/ — generated-site Vue entrypoint and compatibility adapter for the portable research-site runtime.
• booknlp-worker/ — optional isolated BookNLP Document Intelligence worker.
• tests/ — backend, regression, contract, release-consistency, and architecture tests.
• web/tests/frontend/ — Vitest component/domain tests.
• web/tests/e2e/ — Playwright application, Storybook, accessibility, and characterization coverage.
• docs/ — current architecture/domain contracts and historical release notes.
• data/ — local runtime state; git-ignored except placeholders. Never commit its contents.

Documentation

Start with the documents describing current behavior:

• User Guide — feature and workflow reference
• Architecture — runtime boundaries, authority, persistence, and data flow
• cELF 1.0 specification — normative information model and conformance requirements
• Project context — scholarly rationale and implemented-versus-intended capabilities
• GraphQL — read-only cELF query façade
• Realtime — WebSocket notification protocol and resynchronization
• Document Intelligence — derived linguistic analysis and language packs
• Source ingestion — safety, resource limits, and extraction fidelity
• Metadata schemas — configurable field contracts and model guidance
• Metadata memory — reviewed precedents and authority boundaries
• FieldAssertion migration — canonical assertion model and compatibility work
• Code readability — naming, documentation, complex-logic, and refactoring conventions
• Contributing and AGENTS.md — development rules and quality gates

Release history is in CHANGELOG.md and docs/notes/<version>.md. Version-specific release notes are historical records, not current architecture or backlog documents.

License

DerridAI is licensed under the GNU Affero General Public License v3.0.

Copyright © 2026 Aaron John Schlosser, PhD. The application also displays © 2026 The New England Transcendental Club of California.