Semidex

Semidex is an open retrieval and grounded-answering system that turns document collections into searchable, navigable knowledge bases for people, applications, and AI agents. It provides the complete path from document ingestion and structure-aware chunking to dense+sparse embeddings, hybrid retrieval, source evidence, streamed answers, and agent access.

Semidex is designed for personal research, education, internal knowledge, developer tools, and assistants embedded in websites, bots, or other products. The project is an experimental MVP under active development; its current Admin UI is primarily an administration and debugging surface, not a finished user application.

What Semidex provides

• A complete ingestion pipeline: document parsing, deterministic identity, incremental indexing, token-aware chunking, embeddings, and Qdrant storage.
• Structure-aware Markdown: sections, tables, code blocks, and checklists remain connected to canonical source content through skeleton-first chunking.
• Multilingual hybrid retrieval: dense semantic and sparse lexical signals are fused in Qdrant; see retrieval and the benchmark methodology.
• Inspectable evidence: agents can navigate collection skeletons and fetch bounded section or file context instead of treating documents as anonymous fragments.
• Two integration boundaries: a versioned Ask API for applications and a read-only MCP server for independent AI agents.
• Local and cloud composition: providers are selected by deployment rather than embedded into retrieval contracts; see architecture.

Project lines

This repository currently contains the shared Semidex foundation and the local-first reference application from which the editions are composed. Full Semidex does not yet have a supported npm package; run it from source while its installation, UI, and remaining critical functionality are being hardened.

Quick start

Semidex Lite

For a cloud-oriented application integration:

npm install semidex-lite
npx semidex-lite --help

Lite requires a Qdrant Cloud cluster and a supported cloud generation provider. Its package README documents environment variables, indexing, Ask API v1/v2, caller-owned conversation history, and deployment boundaries: packages/lite/README.md.

Full Semidex from source

Prerequisites: Node.js 20.16+, Qdrant Cloud or a local Qdrant server, and Git. Local generation through Ollama and local BGE-M3 embeddings are configured separately.

git clone https://github.com/CodeNoob53/semidex.git
Set-Location semidex
npm install
Copy-Item .env.example .env
npm run admin:build
npm run admin

Open http://127.0.0.1:8642. Before indexing, configure Qdrant and the embedding/generation providers you intend to use.

The complete setup guide covers local and cloud Qdrant, model downloads, Ollama, Windows DirectML/CUDA, indexing, verification, and troubleshooting: Installation guide.

Core commands

PowerShell indexing example:

$env:COLLECTION = 'my-docs'
$env:ONNX_EMBED = '1'
npm run index -- .\docs

Do not mix embedding providers inside one collection. Provider, vector schema, or indexing-schema changes require a compatible reindex.

How it works

documents
  -> parse and preserve source structure
  -> token-aware retrieval chunks + navigation nodes
  -> optional summaries and tags
  -> dense + sparse embeddings
  -> Qdrant vectors, payloads, and collection profile
  -> hybrid retrieval and bounded evidence
       -> Ask API: grounded streamed answers
       -> MCP: search, navigation, and source inspection tools

Markdown currently has the strongest structural support. PDF, office/Pandoc, and plain-text ingestion remain format-dependent; OCR and image understanding are planned rather than implemented. See chunking quality and project status.

Documentation

Documentation entry points: English and Ukrainian.

Status and scope

Semidex has working indexing, Qdrant storage, hybrid retrieval, structural navigation, MCP tools, Ask APIs, Full/Lite composition boundaries, and external retrieval benchmarks. It is not yet a production-ready hosted assistant platform. Authentication for public API exposure, multi-tenancy, finished end-user UX, uniform structural parsing across all formats, OCR, and image understanding remain outside the shipped MVP.

Claims about retrieval quality are tied to the recorded benchmark datasets and profiles; Semidex does not claim general superiority over other RAG systems. See the roadmap for the current snapshot and exit gates.

License and acknowledgements

Semidex is available under the MIT License.

The project is developed with disclosed AI assistance from Claude and OpenAI Codex. Product direction, architectural decisions, review, testing, and final responsibility remain with the author.