djinn-soul/xberg

MIT-era fork of xberg-io/kreuzberg before ELv2 license change

Rust

0

4,339 commits

updated Sep 19, 2026

See the code

README

Kreuzberg

Linkedin- Banner

Extract text, metadata, and code intelligence from 91+ file formats and 248 programming languages at native speeds without needing a GPU.

Key Features

  • Code intelligence – Extract functions, classes, imports, symbols, and docstrings from 248 programming languages via tree-sitter. Results in ExtractionResult.code_intelligence with semantic chunking
  • Extensible architecture – Plugin system for custom OCR backends, validators, post-processors, document extractors, and renderers
  • Polyglot – Native bindings for Rust, Python, TypeScript/Node.js, Ruby, Go, Java, C#, PHP, Elixir, R, and C
  • 91+ file formats – PDF, Office documents, images, HTML, XML, emails, archives, academic formats across 8 categories
  • LLM intelligence – VLM OCR (GPT-4o, Claude, Gemini, Ollama), structured JSON extraction with schema constraints, and provider-hosted embeddings via 146 LLM providers (including local engines: Ollama, LM Studio, vLLM, llama.cpp) through liter-llm
  • OCR support – Tesseract (all bindings, including Tesseract-WASM for browsers), PaddleOCR (all native bindings), EasyOCR (Python), VLM OCR (146 vision model providers including local engines), extensible via plugin API
  • High performance – Rust core with native PDFium, SIMD optimizations and full parallelism
  • Flexible deployment – Use as library, CLI tool, REST API server, or MCP server
  • TOON wire format – Token-efficient serialization for LLM/RAG pipelines, ~30-50% fewer tokens than JSON
  • GFM-quality output – Comrak-based rendering with proper fenced code blocks, table nodes, bracket escaping, and cross-format parity (Markdown, HTML, Djot, Plain)
  • HTML passthrough – HTML-to-Markdown conversion uses html-to-markdown output directly, bypassing lossy intermediate round-trips
  • Memory efficient – Streaming parsers for multi-GB files

Complete Documentation | Live Demo | Installation Guides

Installation

Each language binding provides comprehensive documentation with examples and best practices. Choose your platform to get started:

Scripting Languages:

  • Python – PyPI package, async/sync APIs, OCR backends (Tesseract, PaddleOCR, EasyOCR)
  • Ruby – RubyGems package, idiomatic Ruby API, native bindings
  • PHP – Composer package, modern PHP 8.4+ support, type-safe API, async extraction
  • Elixir – Hex package, OTP integration, concurrent processing
  • R – r-universe package, idiomatic R API, extendr bindings

JavaScript/TypeScript:

  • @kreuzberg/node – Native NAPI-RS bindings for Node.js/Bun, fastest performance
  • @kreuzberg/wasm – WebAssembly for browsers/Deno/Cloudflare Workers, full feature parity (PDF, Excel, OCR, archives)

Compiled Languages:

  • Go – Go module with FFI bindings, context-aware async
  • Java – Maven Central, Foreign Function & Memory API
  • C# – NuGet package, .NET 6.0+, full async/await support

Native:

  • Rust – Core library, flexible feature flags, zero-copy APIs
  • C (FFI) – C header + shared library, pkg-config/CMake support, cross-platform

Containers:

  • Docker – Official images with API, CLI, and MCP server modes (Core: ~1.0-1.3GB, Full: ~1.0-1.3GB with OCR + legacy format support)

Command-Line:

  • CLI – Cross-platform binary, batch processing, MCP server mode

All language bindings include precompiled binaries for both x86_64 and aarch64 architectures on Linux and macOS.

Platform Support

Complete architecture coverage across all language bindings:

LanguageLinux x86_64Linux aarch64macOS ARM64Windows x64
Python✅✅✅✅
Node.js✅✅✅✅
WASM✅✅✅✅
Ruby✅✅✅-
R✅✅✅✅
Elixir✅✅✅✅
Go✅✅✅✅
Java✅✅✅✅
C#✅✅✅✅
PHP✅✅✅✅
Rust✅✅✅✅
C (FFI)✅✅✅✅
CLI✅✅✅✅
Docker✅✅✅-

Note: ✅ = Precompiled binaries available with instant installation. WASM runs in any environment with WebAssembly support (browsers, Deno, Bun, Cloudflare Workers). All platforms are tested in CI. macOS support is Apple Silicon only.

Embeddings Support (Optional)

To use embeddings functionality:

  1. Install ONNX Runtime 1.24+:

  2. Use embeddings in your code - see Embeddings Guide

Note: Kreuzberg requires ONNX Runtime version 1.24+ for embeddings. All other Kreuzberg features work without ONNX Runtime.

Supported Formats

91+ file formats across 8 major categories with intelligent format detection and comprehensive metadata extraction.

Office Documents

CategoryFormatsCapabilities
Word Processing.docx, .docm, .dotx, .dotm, .dot, .odt, .pagesFull text, tables, lists, images, metadata, styles
Spreadsheets.xlsx, .xlsm, .xlsb, .xls, .xla, .xlam, .xltm, .xltx, .xlt, .ods, .numbersSheet data, formulas, cell metadata, charts
Presentations.pptx, .pptm, .ppsx, .potx, .potm, .pot, .keySlides, speaker notes, images, metadata
PDF.pdfText, tables, images, metadata, OCR support
eBooks.epub, .fb2Chapters, metadata, embedded resources
Database.dbfTable data extraction, field type support
Hangul.hwp, .hwpxKorean document format, text extraction

Images (OCR-Enabled)

CategoryFormatsFeatures
Raster.png, .jpg, .jpeg, .gif, .webp, .bmp, .tiff, .tifOCR, table detection, EXIF metadata, dimensions, color space
Advanced.jp2, .jpx, .jpm, .mj2, .jbig2, .jb2, .pnm, .pbm, .pgm, .ppmPure Rust decoders (JPEG 2000, JBIG2), OCR, table detection
Vector.svgDOM parsing, embedded text, graphics metadata

Web & Data

CategoryFormatsFeatures
Markup.html, .htm, .xhtml, .xml, .svgDOM parsing, metadata (Open Graph, Twitter Card), link extraction
Structured Data.json, .yaml, .yml, .toml, .csv, .tsvSchema detection, nested structures, validation
Text & Markdown.txt, .md, .markdown, .djot, .mdx, .rst, .org, .rtfCommonMark, GFM, Djot, MDX, reStructuredText, Org Mode, Rich Text

Email & Archives

CategoryFormatsFeatures
Email.eml, .msgHeaders, body (HTML/plain), attachments, UTF-16 support
Archives.zip, .tar, .tgz, .gz, .7zRecursive extraction, nested archives, metadata

Academic & Scientific

CategoryFormatsFeatures
Citations.bib, .ris, .nbib, .enw, .cslBibTeX/BibLaTeX, RIS, PubMed/MEDLINE, EndNote XML, CSL JSON
Scientific.tex, .latex, .typ, .typst, .jats, .ipynbLaTeX, Typst, JATS journal articles, Jupyter notebooks
Publishing.fb2, .docbook, .dbk, .opmlFictionBook, DocBook XML, OPML outlines
Documentation.pod, .mdoc, .troffPerl POD, man pages, troff

Complete Format Reference →

Code Intelligence (248 Languages)

FeatureDescription
Structure ExtractionFunctions, classes, methods, structs, interfaces, enums
Import/Export AnalysisModule dependencies, re-exports, wildcard imports
Symbol ExtractionVariables, constants, type aliases, properties
Docstring ParsingGoogle, NumPy, Sphinx, JSDoc, RustDoc, and 10+ formats
DiagnosticsParse errors with line/column positions
Syntax-Aware ChunkingSplit code by semantic boundaries, not arbitrary byte offsets

Powered by tree-sitter-language-pack with dynamic grammar download. See TSLP documentation for the full language list.

Key Features

OCR with Table Extraction

Multiple OCR backends (Tesseract, EasyOCR, PaddleOCR) with intelligent table detection and reconstruction. Extract structured data from scanned documents and images with configurable accuracy thresholds.

OCR Backend Documentation →

Batch Processing

Process multiple documents concurrently with configurable parallelism. Optimize throughput for large-scale document processing workloads with automatic resource management.

Batch Processing Guide →

Password-Protected PDFs

Handle encrypted PDFs with single or multiple password attempts. Supports both RC4 and AES encryption with automatic fallback strategies.

PDF Configuration →

Language Detection

Automatic language detection in extracted text using fast-langdetect. Configure confidence thresholds and access per-language statistics.

Language Detection Guide →

Metadata Extraction

Extract comprehensive metadata from all supported formats: authors, titles, creation dates, page counts, EXIF data, and format-specific properties.

Metadata Guide →

AI Coding Assistants

Kreuzberg ships with an Agent Skill that teaches AI coding assistants how to use the library correctly. It works with Claude Code, Codex, Gemini CLI, Cursor, VS Code, Amp, Goose, Roo Code, and any tool supporting the Agent Skills standard.

Install the skill into any project using the Vercel Skills CLI:

npx skills add kreuzberg-dev/kreuzberg

The skill is located at skills/kreuzberg/SKILL.md and is automatically discovered by supported AI coding tools once installed.

Documentation

Contributing

Contributions are welcome! See CONTRIBUTING.md for guidelines.

License

MIT License - see LICENSE for details. You can use Kreuzberg freely in both commercial and closed-source products with no obligations, no viral effects, and no licensing restrictions.

djinn-soul/xberg

MIT-era fork of xberg-io/kreuzberg before ELv2 license change

Rust

0

4,339 commits

updated Sep 19, 2026

See the code

README

Kreuzberg

Linkedin- Banner

Extract text, metadata, and code intelligence from 91+ file formats and 248 programming languages at native speeds without needing a GPU.

Key Features

  • Code intelligence – Extract functions, classes, imports, symbols, and docstrings from 248 programming languages via tree-sitter. Results in ExtractionResult.code_intelligence with semantic chunking
  • Extensible architecture – Plugin system for custom OCR backends, validators, post-processors, document extractors, and renderers
  • Polyglot – Native bindings for Rust, Python, TypeScript/Node.js, Ruby, Go, Java, C#, PHP, Elixir, R, and C
  • 91+ file formats – PDF, Office documents, images, HTML, XML, emails, archives, academic formats across 8 categories
  • LLM intelligence – VLM OCR (GPT-4o, Claude, Gemini, Ollama), structured JSON extraction with schema constraints, and provider-hosted embeddings via 146 LLM providers (including local engines: Ollama, LM Studio, vLLM, llama.cpp) through liter-llm
  • OCR support – Tesseract (all bindings, including Tesseract-WASM for browsers), PaddleOCR (all native bindings), EasyOCR (Python), VLM OCR (146 vision model providers including local engines), extensible via plugin API
  • High performance – Rust core with native PDFium, SIMD optimizations and full parallelism
  • Flexible deployment – Use as library, CLI tool, REST API server, or MCP server
  • TOON wire format – Token-efficient serialization for LLM/RAG pipelines, ~30-50% fewer tokens than JSON
  • GFM-quality output – Comrak-based rendering with proper fenced code blocks, table nodes, bracket escaping, and cross-format parity (Markdown, HTML, Djot, Plain)
  • HTML passthrough – HTML-to-Markdown conversion uses html-to-markdown output directly, bypassing lossy intermediate round-trips
  • Memory efficient – Streaming parsers for multi-GB files

Complete Documentation | Live Demo | Installation Guides

Installation

Each language binding provides comprehensive documentation with examples and best practices. Choose your platform to get started:

Scripting Languages:

  • Python – PyPI package, async/sync APIs, OCR backends (Tesseract, PaddleOCR, EasyOCR)
  • Ruby – RubyGems package, idiomatic Ruby API, native bindings
  • PHP – Composer package, modern PHP 8.4+ support, type-safe API, async extraction
  • Elixir – Hex package, OTP integration, concurrent processing
  • R – r-universe package, idiomatic R API, extendr bindings

JavaScript/TypeScript:

  • @kreuzberg/node – Native NAPI-RS bindings for Node.js/Bun, fastest performance
  • @kreuzberg/wasm – WebAssembly for browsers/Deno/Cloudflare Workers, full feature parity (PDF, Excel, OCR, archives)

Compiled Languages:

  • Go – Go module with FFI bindings, context-aware async
  • Java – Maven Central, Foreign Function & Memory API
  • C# – NuGet package, .NET 6.0+, full async/await support

Native:

  • Rust – Core library, flexible feature flags, zero-copy APIs
  • C (FFI) – C header + shared library, pkg-config/CMake support, cross-platform

Containers:

  • Docker – Official images with API, CLI, and MCP server modes (Core: ~1.0-1.3GB, Full: ~1.0-1.3GB with OCR + legacy format support)

Command-Line:

  • CLI – Cross-platform binary, batch processing, MCP server mode

All language bindings include precompiled binaries for both x86_64 and aarch64 architectures on Linux and macOS.

Platform Support

Complete architecture coverage across all language bindings:

LanguageLinux x86_64Linux aarch64macOS ARM64Windows x64
Python✅✅✅✅
Node.js✅✅✅✅
WASM✅✅✅✅
Ruby✅✅✅-
R✅✅✅✅
Elixir✅✅✅✅
Go✅✅✅✅
Java✅✅✅✅
C#✅✅✅✅
PHP✅✅✅✅
Rust✅✅✅✅
C (FFI)✅✅✅✅
CLI✅✅✅✅
Docker✅✅✅-

Note: ✅ = Precompiled binaries available with instant installation. WASM runs in any environment with WebAssembly support (browsers, Deno, Bun, Cloudflare Workers). All platforms are tested in CI. macOS support is Apple Silicon only.

Embeddings Support (Optional)

To use embeddings functionality:

  1. Install ONNX Runtime 1.24+:

  2. Use embeddings in your code - see Embeddings Guide

Note: Kreuzberg requires ONNX Runtime version 1.24+ for embeddings. All other Kreuzberg features work without ONNX Runtime.

Supported Formats

91+ file formats across 8 major categories with intelligent format detection and comprehensive metadata extraction.

Office Documents

CategoryFormatsCapabilities
Word Processing.docx, .docm, .dotx, .dotm, .dot, .odt, .pagesFull text, tables, lists, images, metadata, styles
Spreadsheets.xlsx, .xlsm, .xlsb, .xls, .xla, .xlam, .xltm, .xltx, .xlt, .ods, .numbersSheet data, formulas, cell metadata, charts
Presentations.pptx, .pptm, .ppsx, .potx, .potm, .pot, .keySlides, speaker notes, images, metadata
PDF.pdfText, tables, images, metadata, OCR support
eBooks.epub, .fb2Chapters, metadata, embedded resources
Database.dbfTable data extraction, field type support
Hangul.hwp, .hwpxKorean document format, text extraction

Images (OCR-Enabled)

CategoryFormatsFeatures
Raster.png, .jpg, .jpeg, .gif, .webp, .bmp, .tiff, .tifOCR, table detection, EXIF metadata, dimensions, color space
Advanced.jp2, .jpx, .jpm, .mj2, .jbig2, .jb2, .pnm, .pbm, .pgm, .ppmPure Rust decoders (JPEG 2000, JBIG2), OCR, table detection
Vector.svgDOM parsing, embedded text, graphics metadata

Web & Data

CategoryFormatsFeatures
Markup.html, .htm, .xhtml, .xml, .svgDOM parsing, metadata (Open Graph, Twitter Card), link extraction
Structured Data.json, .yaml, .yml, .toml, .csv, .tsvSchema detection, nested structures, validation
Text & Markdown.txt, .md, .markdown, .djot, .mdx, .rst, .org, .rtfCommonMark, GFM, Djot, MDX, reStructuredText, Org Mode, Rich Text

Email & Archives

CategoryFormatsFeatures
Email.eml, .msgHeaders, body (HTML/plain), attachments, UTF-16 support
Archives.zip, .tar, .tgz, .gz, .7zRecursive extraction, nested archives, metadata

Academic & Scientific

CategoryFormatsFeatures
Citations.bib, .ris, .nbib, .enw, .cslBibTeX/BibLaTeX, RIS, PubMed/MEDLINE, EndNote XML, CSL JSON
Scientific.tex, .latex, .typ, .typst, .jats, .ipynbLaTeX, Typst, JATS journal articles, Jupyter notebooks
Publishing.fb2, .docbook, .dbk, .opmlFictionBook, DocBook XML, OPML outlines
Documentation.pod, .mdoc, .troffPerl POD, man pages, troff

Complete Format Reference →

Code Intelligence (248 Languages)

FeatureDescription
Structure ExtractionFunctions, classes, methods, structs, interfaces, enums
Import/Export AnalysisModule dependencies, re-exports, wildcard imports
Symbol ExtractionVariables, constants, type aliases, properties
Docstring ParsingGoogle, NumPy, Sphinx, JSDoc, RustDoc, and 10+ formats
DiagnosticsParse errors with line/column positions
Syntax-Aware ChunkingSplit code by semantic boundaries, not arbitrary byte offsets

Powered by tree-sitter-language-pack with dynamic grammar download. See TSLP documentation for the full language list.

Key Features

OCR with Table Extraction

Multiple OCR backends (Tesseract, EasyOCR, PaddleOCR) with intelligent table detection and reconstruction. Extract structured data from scanned documents and images with configurable accuracy thresholds.

OCR Backend Documentation →

Batch Processing

Process multiple documents concurrently with configurable parallelism. Optimize throughput for large-scale document processing workloads with automatic resource management.

Batch Processing Guide →

Password-Protected PDFs

Handle encrypted PDFs with single or multiple password attempts. Supports both RC4 and AES encryption with automatic fallback strategies.

PDF Configuration →

Language Detection

Automatic language detection in extracted text using fast-langdetect. Configure confidence thresholds and access per-language statistics.

Language Detection Guide →

Metadata Extraction

Extract comprehensive metadata from all supported formats: authors, titles, creation dates, page counts, EXIF data, and format-specific properties.

Metadata Guide →

AI Coding Assistants

Kreuzberg ships with an Agent Skill that teaches AI coding assistants how to use the library correctly. It works with Claude Code, Codex, Gemini CLI, Cursor, VS Code, Amp, Goose, Roo Code, and any tool supporting the Agent Skills standard.

Install the skill into any project using the Vercel Skills CLI:

npx skills add kreuzberg-dev/kreuzberg

The skill is located at skills/kreuzberg/SKILL.md and is automatically discovered by supported AI coding tools once installed.

Documentation

Contributing

Contributions are welcome! See CONTRIBUTING.md for guidelines.

License

MIT License - see LICENSE for details. You can use Kreuzberg freely in both commercial and closed-source products with no obligations, no viral effects, and no licensing restrictions.