Vectorizer

High-performance vector database and search engine in Rust for semantic search, document indexing, and AI applications. Ships as a Cargo workspace (5 crates) with binary RPC + HTTP transports, a React dashboard, and native SDKs for Rust, Python, TypeScript, Go, and C#.

✨ Key Features

Transport & API

• VectorizerRPC (default, port 15503) — binary MessagePack over TCP, multiplexed connection pool. See wire spec.
• REST API (port 15002) — universal HTTP fallback, powers the dashboard and any caller that doesn't speak raw TCP.
• gRPC — Qdrant-compatible service.
• GraphQL — full REST parity with async-graphql + GraphiQL playground.
• MCP — 31 focused tools for AI model integration (Cursor, Claude Desktop, etc.).
• UMICP Protocol — native JSON types + tool discovery endpoint.

Performance

• SIMD acceleration — AVX2-optimized vector ops with runtime CPU detection (5-10x faster).
• Metal GPU — macOS Apple Silicon via hive-gpu 0.2; logs render real device name, driver, VRAM.
• Sub-3ms search (CPU) / <1ms (GPU) via HNSW indexing.
• 4-5x faster than Qdrant in head-to-head benchmarks (0.16-0.23ms vs 0.80-0.87ms avg latency).

Storage

• .vecdb unified format — 20-30% space savings, automatic snapshots.
• Memory-mapped storage — datasets larger than RAM, efficient OS paging.
• Product Quantization — 64x memory reduction with minimal accuracy loss.
• Scalar Quantization + cache hit ratio metrics.

High Availability & Scaling

• Raft consensus via openraft (pinned =0.10.0-alpha.30) — automatic leader election in 1-5s, write-redirect via HTTP 307, WAL-backed durable replication, DNS discovery for Kubernetes headless services.
• Master-Replica — TCP streaming replication with full/partial sync, exponential reconnect backoff (5s→60s).
• Distributed sharding — horizontal scaling with automatic routing; distributed hybrid search via RemoteHybridSearch RPC with dense-only fallback for mixed-version clusters.
• HiveHub cluster mode — multi-tenant with quotas, usage tracking, tenant isolation, mandatory MMap storage, 1GB cache cap.

Search

• Semantic similarity — Cosine, Euclidean, Dot Product.
• Hybrid search — Dense + Sparse with Reciprocal Rank Fusion (RRF).
• Intelligent search — query expansion, semantic reranking.
• Multi-collection search across projects.
• Graph relationships — automatic edge discovery, neighbor exploration, shortest-path finding.

Embeddings & Docs

• Built-in providers — TF-IDF, BM25, FastEmbed, BERT, MiniLM, custom models. embedding_provider (on POST /collections) and model (on POST /embed) are honoured contracts as of v3.4.0 (issue #306) — unknown providers / models return 400 unsupported_provider / 400 unsupported_model with the available list; no more silent BM25-512 coercion. Discover registered providers via GET /stats.providers or the list_providers MCP tool. See docs/users/guides/EMBEDDINGS.md.
• Document conversion — PDF, DOCX, XLSX, PPTX, HTML, XML, images (14 formats).
• Qdrant API compatibility — Snapshots, Sharding, Cluster Management, Query (with prefetch), Search Groups, Matrix, Named Vectors (partial), PQ/Binary quantization config.
• Summarization — extractive, keyword, sentence, abstractive (OpenAI GPT).

Security

• JWT + API Key authentication with RBAC, scoped API keys (per-collection permissions), atomic key rotation with grace window, RFC 7662 token introspection, admin audit log (in-memory ring + daily-rotated JSONL).
• API key usage metrics — per-key usage_count (atomic, lock-free) plus a 30-day per-day ring buffer surfaced via GET /auth/keys/{id}/usage.
• Permission update without rotation — PUT /auth/keys/{id}/permissions swaps permissions/scopes while keeping key_hash/id/user_id/created_at immutable.
• Hardened dashboard cookies + CSRF — login/refresh emit vectorizer_session (HttpOnly; Secure; SameSite=Strict) plus a sibling XSRF-TOKEN cookie; require_csrf_middleware guards every mutating /auth/* and /admin/* request. auth.cookies.insecure_dev opt-out for plain-HTTP loopback dev (boot rejects it on 0.0.0.0).
• Loopback dev-mode auth bypass — auth.dev_mode_skip_loopback short-circuits credential validation as local-dev-admin and stamps X-Vectorizer-Dev-Mode: true on every response. Boot fails on any non-loopback host.
• JWT secret is mandatory — boot refuses to start with empty / default / <32 char secrets when auth is enabled.
• First-run root credentials written to {data_dir}/.root_credentials (0o600), never logged.
• Payload encryption — optional ECC-P256 + AES-256-GCM, zero-knowledge, per-collection policies (docs).
• TLS 1.2/1.3 with mTLS, configurable cipher suites, ALPN.
• Per-API-key rate limiting with tiers + overrides.
• Path-traversal guard on file discovery; canonicalized base, symlink-escape refusal.

UI

• Web Dashboard — React + TypeScript; JWT login, graph CRUD (edges, neighbors, paths), collection management, API sandbox, setup wizard with glassmorphism design. Embedded in the binary (~26MB, no external assets needed).
• Desktop GUI — Electron + vis-network for visual database management.

🎉 Latest Release: v3.5.0

Highlights — see CHANGELOG.md for the full breakdown.

Fixed — Text search survived nothing: BM25 vocabulary was never restored after a restart (phase37)

• Auto-save wrote a stub tokenizer (vocab_size: 0) and no code path ever reloaded the vocabulary, so a restarted server embedded queries in a hash-fallback space disjoint from the stored vectors — text search returned zero hits until a full re-index.
• Auto-save now persists the real vocabulary and bootstrap restores the newest snapshot (from raw files or from inside vectorizer.vecdb) into the query-time provider. Collections without a usable snapshot are flagged degraded_vocabulary:{name} and logged instead of silently degrading. Pinned by an end-to-end save → restart → search test.

Fixed — WAL durability (phase37)

• The WAL only flush()ed — acknowledged writes died with the OS page cache on power loss. It now fsyncs on every append/checkpoint (wal.fsync, default on), frames each record with CRC32 + length (a torn final record no longer aborts recovery of everything after it), and sequence numbers survive restarts and concurrent transactions without duplication.

Fixed — bulk_update_metadata deadlocked on every call (phase39)

• The handler held a DashMap shard reference across an operation that takes a write reference on the same shard. The endpoint had zero test coverage, so every production call simply hung forever. Caught by the first-ever run of the new in-process handler coverage; the deadlock class was removed in phase41 (VectorStore::update no longer takes an unconditional shard write ref).

Performance — search no longer blocks behind batch inserts (phase38)

• insert_batch held the HNSW index write lock for the entire batch, collapsing search p99 under mixed load. Searches now proceed against the internally-synchronized index while writers serialize on a dedicated mutex. Per-vector copies on the insert path cut from 3-4 to 1.
• Product Quantization and Binary quantization are actually reachable (the integration silently substituted 8-bit scalar for everything before); AVX2 + NEON quantize/dequantize and AVX2 int8 dot-product kernels landed, CI-verified against the scalar oracle on x86_64 and native arm64.

Security — Docker image CVE posture (phase35)

• 3.4.0 shipped with 31 base-image CVEs (4 HIGH in openssl on the TLS hot path). 3.5.0 rebuilds against the patched dhi.io/debian-base:trixie pinned by digest: 0 CRITICAL / 0 HIGH; the unfixable remainder (no patched version exists in Debian trixie) is documented per-CVE in an OpenVEX attestation attached to the image, so Scout dashboards read clean without hiding real signal. A weekly + on-release CI gate fails on any new fixable CRITICAL/HIGH and tracks base-digest staleness.

Changed — API parity + hardening (phase40)

• MCP now mirrors REST: delete_collection, embed_text, contextual_search, get_database_stats, the 8-op discovery pipeline, and batch_* tools; MCP error codes are mapped (not-found is no longer a generic internal error); RPC error frames carry the stable error code.
• Search limit (and hybrid dense_k/sparse_k/final_k) is clamped server-side at 100 — the schema said so, the handlers now enforce it.
• GraphQL multi-tenancy bug fixed: upload_file and create_collection disagreed on the tenant prefix, so uploads landed in a different collection than the one created.
• First boot with the shipped default config now works: an empty jwt_secret auto-generates a persisted secret (with a prominent warning) instead of failing the bind check. Unknown config keys warn at boot; the config is loaded once through the layered loader so mode overrides actually reach every subsystem.

Testing (phase39) — an in-process REST harness runs the real production router with no server process: 98 formerly live-server-only tests plus coverage for ~30 previously untested handlers now run on every PR; a CI gate keeps the #[ignore] count from creeping; a weekly job boots the server and runs all four SDK integration suites against it.

Build — dependency refresh (phase36) — rmcp 2.1 (MCP 2025-11-25), candle 0.11, aes-gcm 0.11, ~120 compatible bumps, SDK dep refreshes, and dependabot now covers every SDK ecosystem.

Server-side at v3.5.0; all five SDKs synced to v3.5.0.

---

v3.4.0 highlights (previous release)

Fixed — Container deployments lose all collections on restart (phase32, #300)

• 3.3.0 image wrote persistent state to /.local/share/vectorizer/ even though the README advertised /data as the volume mount. The XDG path lived on the container's writable layer, so docker compose up -d --force-recreate vectorizer silently wiped every collection.
• Image defaults VECTORIZER_DATA_DIR=/data and seeds the directory in the writable-dirs stage with the nonroot user as owner. A single --volume vec-data:/data mount now captures collections, auth keys, JWT secret, and snapshots.
• vectorizer --data-dir <path> is a first-class CLI flag; resolves through vectorizer_core::paths::data_dir so every persistence subpath (auth, vector store, snapshots, fastembed cache) picks up the override.
• Startup emits WARN data dir at <path> is ephemeral; recommend mounting a volume when the resolved data dir has no backing mount (Linux only, via /proc/self/mountinfo). Surfaces the trap on the first boot without a volume.
• Migration runbook in docs/users/configuration/DATA_DIRECTORY.md for operators who mounted the workaround /.local/share/vectorizer second volume.

Added — Honour embedding_provider / model in REST contracts (phase33, #306) — contract change

• 3.3.0 silently coerced every embedding_provider to BM25-512. Downstream consumers (e.g. hivellm/cortex) saw their hybrid pipeline degrade to keyword-only because the vector lane was returning lexical BM25 vectors regardless of what they posted.
• POST /collections honours embedding_provider. Unknown provider → 400 unsupported_provider { requested, available }. Caller-requested dimension that conflicts with the provider's native dimension → 400 provider_dimension_mismatch.
• POST /embed honours model. Unknown model → 400 unsupported_model { requested, available }. Response echoes the resolved model so callers can confirm which provider produced the vector.
• GET /stats lists providers[] + default_provider so clients can discover the registered embedding surface without trial-and-error. Mirrored as the list_providers MCP tool.
• CollectionConfig.embedding_provider: String persists which provider the collection was created with. Legacy .vecdb files default to "bm25" via serde so reload stays lossless.
• Bootstrap registers every available provider at boot — without this, POST /collections {embedding_provider: "bm25"} would have returned 400 on any fastembed-default deployment.
• Optional FastEmbed Docker variant: docker build --build-arg ENABLE_FASTEMBED=1 --build-arg NO_DEFAULT_FEATURES=0 --build-arg FEATURES=fastembed . ships an image with all-MiniLM-L6-v2 (384-dim dense) registered alongside bm25. Default published image stays slim (BM25-only).

Fixed — Bumped vulnerable transitive deps

• axios <1.16.0 → 1.17.0 (gui pnpm override): closes 6 dependabot alerts (proxy-auth leak via redirects, prototype-pollution MITM via config.proxy, shouldBypassProxy IPv4-mapped IPv6 NO_PROXY bypass, header injection via merge gadgets, null-prototype patch bypass).
• tmp <0.2.6 → 0.2.7 (gui pnpm override): closes path-traversal via unsanitized prefix/postfix.
• react-router 7.14.2 → 7.17.0 (dashboard top-level + override): closes DoS via unbounded path expansion in __manifest.
• tar 0.4.45 → 0.4.46 (root Cargo.lock): closes PAX header desynchronization.

Build

• Docker builder base bumped from lukemathwalker/cargo-chef:rust-1.90-bookworm to rust:1.95-slim-trixie. glibc 2.40 matches the runtime dhi.io/debian-base:trixie, clearing the __isoc23_strtol/__isoc23_strtoull link errors that surfaced when fastembed pulled in ORT prebuilt binaries linked against glibc 2.38+ symbols.
• libstdc++.so.6 copied from the builder into the runtime stage so the fastembed Docker variant boots without error while loading shared libraries: libstdc++.so.6.

Server-side at v3.4.0. The Rust SDK tracks server versioning; TypeScript, Python, Go, and C# SDKs are also on v3.4.0 (no breaking server-contract changes beyond the embedding-provider error shapes documented above).

---

For prior releases (v3.3.0 and earlier) see CHANGELOG.md.

🚀 Quick Start

Install Script (Linux/macOS)

curl -fsSL https://raw.githubusercontent.com/hivellm/vectorizer/main/scripts/install.sh | bash

Installs CLI + systemd service. Commands: sudo systemctl {status|restart|stop} vectorizer, sudo journalctl -u vectorizer -f.

Install Script (Windows)

powershell -c "irm https://raw.githubusercontent.com/hivellm/vectorizer/main/scripts/install.ps1 | iex"

Installs CLI + Windows Service (requires Admin). Commands: Get-Service Vectorizer, {Start|Stop|Restart}-Service Vectorizer.

Docker

docker run -d \
  --name vectorizer \
  -p 15002:15002 -p 15503:15503 \
  -v vec-data:/data \
  -e VECTORIZER_AUTH_ENABLED=true \
  -e VECTORIZER_ADMIN_USERNAME=admin \
  -e VECTORIZER_ADMIN_PASSWORD=your-secure-password \
  -e VECTORIZER_JWT_SECRET=$(openssl rand -hex 64) \
  --restart unless-stopped \
  hivehub/vectorizer:latest

Starting in hivehub/vectorizer:3.4.0 the image defaults VECTORIZER_DATA_DIR=/data, so a single --volume vec-data:/data mount captures every collection, auth key, JWT secret, and snapshot. docker compose up -d --force-recreate vectorizer is now safe; see docs/users/configuration/DATA_DIRECTORY.md for the 3.3.0 migration runbook (issue #300).

Docker Compose with profiles:

cp config/.env.example .env
# Edit .env with your credentials
docker compose --profile default up -d          # standalone
docker compose --profile dev up -d              # dev overlay
docker compose --profile ha up -d               # Raft cluster
docker compose --profile hub up -d              # multi-tenant

Profiles are mutually exclusive on host port 15002.

Images: Docker Hub · GHCR

Build from Source

git clone https://github.com/hivellm/vectorizer.git
cd vectorizer

cargo build --release                          # Basic
cargo build --release --features hive-gpu      # macOS Metal
cargo build --release --features full          # All features
./target/release/vectorizer

Keeping target/ bounded. Cargo never GCs target/ — stale rlibs accumulate until you nuke them. This repo already pins [profile.dev] debug = "line-tables-only" + [profile.release] strip = true + CARGO_INCREMENTAL=0 on CI; on a developer box, run bash scripts/sweep-target.sh (or pwsh scripts/sweep-target.ps1 on Windows) weekly to drop stale artifacts without losing the incremental hot set. Full runbook + scheduler examples in docs/development/rust-target-hygiene.md (issue #320).

Access Points

Upgrading from v2.x? RPC is now on by default on port 15503. REST is unchanged. If you can't expose the new port, set rpc.enabled: false. See v3.x migration guide.

Configuration

Configs live under config/:

config/
├── config.yml             # Base config (your deployment)
├── config.example.yml     # Reference
├── modes/
│   ├── dev.yml            # Layered override: verbose logs, loopback, watcher on
│   └── production.yml     # Layered override: warn logs, larger threads/cache, zstd, scheduled snapshots
└── presets/               # Standalone full configs (legacy style)
    ├── production.yml
    ├── cluster.yml
    ├── hub.yml
    └── development.yml

Layered loader (recommended):

VECTORIZER_MODE=production ./target/release/vectorizer

Merges config/modes/production.yml over config/config.yml. Typos in the mode override fail fast at boot.

Authentication

Auth is enabled by default in Docker. Default creds — change in production.

# Login
curl -X POST http://localhost:15002/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"admin"}'

# JWT in requests
curl http://localhost:15002/collections \
  -H "Authorization: Bearer YOUR_JWT_TOKEN"

# Create API key (JWT required)
curl -X POST http://localhost:15002/auth/keys \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_JWT_TOKEN" \
  -d '{"name":"Production","permissions":["read","write"],"expires_in_days":90}'

# API key in requests (NO Bearer prefix)
curl http://localhost:15002/collections \
  -H "Authorization: YOUR_API_KEY"

Production must set:

• VECTORIZER_JWT_SECRET — ≥32 chars, not the historical default. Boot aborts otherwise.
• VECTORIZER_ADMIN_PASSWORD — strong, ≥32 chars.

First-run root credentials are written to {data_dir}/.root_credentials (0o600), never printed to stdout. Read and delete after first login.

See Docker Authentication Guide and Security Policy.

📊 Performance

Benchmark vs Qdrant

• Search: 4-5x faster (0.16-0.23ms vs 0.80-0.87ms avg latency).
• Insert: Fire-and-forget pattern, configurable batch / body limits, background processing.
• Scenarios: Small (1K) / Medium (5K) / Large (10K) vectors × dimensions 384 / 512 / 768.

See Benchmark Documentation.

🔄 Feature Comparison

Key Differentiators

• MCP integration (31 tools) — native AI-agent protocol.
• Graph relationships — auto-discovery + full GUI (edges, path-finding, neighbor exploration).
• GraphQL — full REST parity + GraphiQL.
• Document processing — 14 formats built in.
• Qdrant compatibility — full API + migration tools.
• Performance — 4-5x faster than Qdrant in benchmarks.
• Binary RPC default — MessagePack over TCP on port 15503 for low-overhead client traffic.
• Complete SDK coverage — Rust, Python, TypeScript (+JS), Go, C# — all on v3.5.0.

Best fit: AI apps needing MCP, document ingestion, graph relationships, and sub-ms search with an embedded dashboard.

🎯 Use Cases

• RAG systems — semantic search with automatic document conversion.
• Document search — PDFs, Office, web content.
• Code analysis — semantic code navigation.
• Knowledge bases — enterprise multi-format search.

🔧 MCP Integration

Cursor / Claude Desktop config:

{
  "mcpServers": {
    "vectorizer": {
      "url": "http://localhost:15002/mcp",
      "type": "streamablehttp"
    }
  }
}

Available Tools (31)

Core operations (9) list_collections · create_collection · get_collection_info · insert_text · get_vector · update_vector · delete_vector · search · multi_collection_search

Advanced search (4) search_intelligent (query expansion) · search_semantic (reranking) · search_extra (combined) · search_hybrid (dense + sparse RRF)

Discovery & files (7) filter_collections · expand_queries · get_file_content · list_files · get_file_chunks · get_project_outline · get_related_files

Graph (8) graph_list_nodes · graph_get_neighbors · graph_find_related · graph_find_path · graph_create_edge · graph_delete_edge · graph_discover_edges · graph_discover_status

Maintenance (3) list_empty_collections · cleanup_empty_collections · get_collection_stats

Cluster-management operations are REST-only for security.

📦 Client SDKs

Server-side at v3.5.0. The Rust SDK tracks server versioning; the TypeScript, Python, Go, and C# SDKs are also on v3.5.0. The TypeScript SDK ships compiled CJS + ESM — usable from plain JavaScript, no separate JS package needed.

Every SDK accepts both vectorizer://host[:port] (RPC, default port 15503) and http(s)://host[:port] (REST) URLs through the same endpoint parser.

🔄 Qdrant Migration

• Config migration — parse Qdrant YAML/JSON → Vectorizer format.
• Data migration — export from Qdrant, import into Vectorizer.
• Validation — integrity + compatibility checks.
• REST compatibility — full Qdrant API at /qdrant/*.

use vectorizer::migration::qdrant::{QdrantDataExporter, QdrantDataImporter};

let exported = QdrantDataExporter::export_collection(
    "http://localhost:6333",
    "my_collection"
).await?;

let result = QdrantDataImporter::import_collection(&store, &exported).await?;

See Qdrant Migration Guide.

☁️ HiveHub Cloud

Multi-tenant cluster mode integration with HiveHub.Cloud.

• Tenant isolation — owner-scoped collections.
• Quota enforcement — collections / vectors / storage per tenant.
• Usage tracking — automatic reporting.
• User-scoped backups.

hub:
  enabled: true
  api_url: "https://api.hivehub.cloud"
  tenant_isolation: "collection"
  usage_report_interval: 300

export HIVEHUB_SERVICE_API_KEY="your-service-api-key"

Cluster-mode requirements (enforced at boot):

cluster:
  enabled: true
  node_id: "node-1"
  memory:
    max_cache_memory_bytes: 1073741824
    enforce_mmap_storage: true
    disable_file_watcher: true
    strict_validation: true

See HiveHub Integration and Cluster Memory Limits.

🏗️ Workspace Layout

crates/
├── vectorizer-core/       # Foundation: error, codec, quantization, simd, compression, paths
├── vectorizer-grpc/       # tonic-generated gRPC types (first-party, cluster, Qdrant-compatible)
├── vectorizer/            # Engine (umbrella): db, embedding, models, cache, persistence, search, ...
├── vectorizer-server/     # Transport: HTTP / gRPC / MCP / RPC + binary
└── vectorizer-cli/        # CLI binaries
sdks/rust/                 # Rust SDK — RPC transport via the shared `thunder-rpc` crate

Runtime directories resolve to platform-standard locations (~/.local/share/vectorizer/ on Linux, ~/Library/Application Support/vectorizer/ on macOS, %APPDATA%\vectorizer\ on Windows), overridable via VECTORIZER_DATA_DIR / VECTORIZER_LOGS_DIR.

📚 Documentation

• User Documentation — install + tutorials
• API Reference — REST
• VectorizerRPC Spec — wire protocol
• RPC Operator Guide
• Configuration — layered loader
• Bulk-upsert Backpressure Runbook — 429 / Retry-After, vocab-build cap, ops metrics (#263)
• v3.x Migration — RPC-default rollout
• Dashboard Integration
• Qdrant Compatibility
• HiveHub Integration
• Cluster Memory Limits
• MCP Guide
• Encryption
• Technical Specs — architecture, performance, implementation

📄 License

Apache License 2.0 — see LICENSE.

🤝 Contributing

See CONTRIBUTING.md.