based_vector

A basic vector database for Rust with multiple similarity engines and ONNX-powered embedding generation.

Features

• Multiple Similarity Methods: Choose between cosine similarity or custom MLP-based learned similarity
• ONNX Integration: Built-in embedding generation using sentence transformers from Hugging Face
• File-Backed Storage: Persistent storage with atomic writes in .based_vector/ directory
• CLI and Library: Use as a command-line tool or integrate into your Rust application
• Metadata Filtering: Search with custom metadata filters
• Type-Safe: Generic over data types with full serialization support

Installation

Add to your Cargo.toml:

[dependencies]
based_vector = "0.1"

Or install the CLI:

cargo install based_vector

Quick Start - CLI

Initialize a database

# Create a new database in the current directory
based_vector init

# Or specify similarity method
based_vector init --method mlp

Insert data

# Insert a document
based_vector insert --text "Your document text here"

# Insert with metadata
based_vector insert \
  --text "Rust is a systems programming language" \
  --metadata category=programming \
  --metadata language=rust

Search

# Basic search (default top-5)
based_vector search --query "programming languages"

# Search with specific top-k
based_vector search --query "programming languages" --top-k 10

# Search with threshold (returns all results above threshold)
based_vector search --query "rust" --threshold 0.7

# Combine threshold + top-k (at most 10 results with score >= 0.7)
based_vector search --query "rust" --top-k 10 --threshold 0.7

# JSON output
based_vector search --query "rust" --format json

Other commands

# List entries
based_vector list --limit 10

# Show statistics
based_vector stats

# Delete an entry
based_vector delete --id <entry-id>

# Export to JSON
based_vector export --output backup.json

# Import from JSON
based_vector import --input backup.json

Quick Start - Library

use based_vector::{Db, Document, Embeddable};
use based_vector::embedder::OnnxEmbedder;
use based_vector::cosine::CosineSimilarityEngine;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize ORT with quiet logging (optional but recommended)
    based_vector::init_ort_quiet();

    // Create a new database
    let mut db: Db<Document> = Db::new();

    // Initialize embedder (downloads model on first run)
    let mut embedder = OnnxEmbedder::new()?;
    embedder.initialize().await?;

    // Insert documents
    let doc1 = Document::new("Rust is a systems programming language");
    let embedding1 = embedder.embed_mut(&doc1.embedding_text())?;
    db.insert_with_embedding(doc1, embedding1)?;

    let doc2 = Document::new("Python is great for data science");
    let embedding2 = embedder.embed_mut(&doc2.embedding_text())?;
    db.insert_with_embedding(doc2, embedding2)?;

    // Search
    let query = "programming languages";
    let query_embedding = embedder.embed_mut(query)?;
    let engine = CosineSimilarityEngine::new();

    // Top-K search
    let results = db.search(&query_embedding, 5, &engine)?;

    for result in results {
        println!("Score: {:.4}, Text: {}", result.score, result.entry.data.text);
    }

    // Or threshold-based search (all results above 0.7 similarity)
    let threshold_results = db.search_with_threshold(&query_embedding, 0.7, &engine)?;
    println!("Found {} results above threshold", threshold_results.len());

    // Save to disk
    db.save(None::<&str>)?;

    Ok(())
}

Custom Data Types

You can use custom types by implementing the Embeddable trait:

use based_vector::{Db, Embeddable};
use serde::{Deserialize, Serialize};

#[derive(Debug, Clone, Serialize, Deserialize)]
struct Article {
    title: String,
    content: String,
    author: String,
}

impl Embeddable for Article {
    fn embedding_text(&self) -> String {
        // Decide what text to embed
        format!("{}\n\n{}", self.title, self.content)
    }
}

fn main() {
    let mut db: Db<Article> = Db::new();
    // Use db with Articles...
}

Architecture

Modules

• embedder: ONNX-based embedding generation with Hugging Face models
• cosine: Cosine similarity engine (fast, lightweight)
• mlp: MLP-based learned similarity (requires trained ONNX model)
• storage: File-backed persistence with atomic writes
• query: Search and filtering functionality
• config: Configuration and builder patterns

Storage Format

The database is stored in a .based_vector/ directory:

.based_vector/
├── entries.bin      # Serialized entries (bincode)
├── config.json      # Database metadata
└── index.bin        # Optional index structures (future)

Embedding Models

The default model is sentence-transformers/all-MiniLM-L6-v2 (384 dimensions).

Supported models:

• sentence-transformers/all-MiniLM-L6-v2 (default)
• sentence-transformers/all-mpnet-base-v2
• sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
• sentence-transformers/distiluse-base-multilingual-cased-v2

Models are cached in ~/.cache/based_vector/models/ (or OS equivalent).

MLP Similarity

To use MLP-based similarity, you need a trained ONNX model. The model should:

1. Accept features: concat(embed1, embed2) + abs_diff + element_wise_product
2. Output a single similarity score (float)

Place your model at ~/.cache/based_vector/models/similarity_mlp.onnx or specify a custom path.

Performance

• Embedding generation: ~10-50ms per document (depends on model and hardware)
• Cosine similarity: O(n*d) where n=entries, d=dimensions
• Storage: Compact bincode format with optional compression

Requirements

• Rust 1.70+
• ONNX Runtime (automatically downloaded)
• ~500MB disk space for models (on first download)

Development

# Build
cargo build --release

# Run tests
cargo test

# Run with verbose logging (both app and ONNX Runtime)
RUST_LOG=debug ORT_LOG_LEVEL=1 cargo run -- search --query "test"

# App verbose, ORT quiet (default)
RUST_LOG=debug cargo run -- search --query "test"

Controlling ONNX Runtime Logging

By default, based_vector attempts to suppress verbose ONNX Runtime logs by setting ORT_LOG_LEVEL=3 (Error level).

If you still see verbose ORT logs, set the environment variable before running:

# Windows (PowerShell)
$env:ORT_LOG_LEVEL="3"
based_vector search --query "test"

# Windows (CMD)
set ORT_LOG_LEVEL=3
based_vector search --query "test"

# Linux/macOS
export ORT_LOG_LEVEL=3
based_vector search --query "test"

# Or inline
ORT_LOG_LEVEL=3 based_vector search --query "test"

ORT_LOG_LEVEL values:

• 0 = Verbose (very noisy - shows all optimization steps)
• 1 = Info (shows hardware detection, graph transformations)
• 2 = Warning
• 3 = Error (default - quiet)
• 4 = Fatal (completely silent)

To see ONNX details for debugging:

ORT_LOG_LEVEL=1 based_vector search --query "test"

Limitations

• Linear search only (no HNSW index yet)
• Single-threaded ONNX inference per embedder instance
• EmbeddingEngine trait requires mutable access due to ONNX session

Roadmap

• HNSW index for faster similarity search
• Batch embedding optimization
• Custom tokenizer support
• Database migrations
• Async storage operations
• Compression support
• SVM similarity engine

License

MIT OR Apache-2.0

Contributing

Contributions welcome! Please open an issue or PR.

Acknowledgments

• Built with ort for ONNX Runtime
• Uses tokenizers for text tokenization
• Models from Hugging Face