VaireDB

VaireDB is a cloud native, distributed SQL database that combines PostgreSQL wire compatibility, with DuckDB's columnar vectorized execution engine for high-throughput analytical workloads across a horizontally scalable cluster.

Read the When use VaireDB section for understanding where VaireDB is best suited for your use cases.

[!IMPORTANT]
VaireDB is currently under active development. Breaking changes may occur and major features may be added. Consider this a work in progress and not yet ready for production use. Any contributions are welcome for targeting the production-ready v1.0 release.

Overview

VaireDB exposes a unified SQL interface through the PostgreSQL wire protocol (v3), allowing connections from any standard PostgreSQL client (psql, JDBC, etc.). Under the hood, data is hash-sharded across core nodes, each running an embedded DuckDB instance optimized for OLAP queries.

Key characteristics:

• PostgreSQL-compatible SQL via DataFusion parser with automatic dialect translation to DuckDB.
• Horizontal scalability through hash sharding across core nodes (range sharding planned).
• Analytical performance powered by DuckDB's columnar vectorized engine.
• Fault tolerance with configurable replication factor and quorum-based writes.
• Column pseudonymization for compliance: declared columns are HMAC-SHA256 hashed in the coordinator so plaintext never reaches storage.
• Operational simplicity as self-contained Rust binaries with YAML configuration.

When use VaireDB

Use Cases

• CQRS read side — an analytical database positioned close to microservices' transactional databases, serving the query (read) side of a CQRS architecture. Transactional systems keep handling writes in their own stores, while VaireDB absorbs the heavy read and aggregation traffic that would otherwise contend with operational workloads — keeping write paths fast and read paths scalable.

• Shared read model — a denormalized, read-optimized view of data shared across a microservices ecosystem. Instead of each service repeatedly joining and reshaping data from many sources, VaireDB holds a consolidated, query-friendly representation that teams can reuse, reducing duplicated effort and keeping cross-service reporting consistent.

• Data fabric or micro-fabric — deployable both as a wide data fabric spanning the whole ecosystem and as a micro-fabric local to each bounded context. The same engine scales from a single bounded context to an organization-wide layer, so you can start small within one domain and grow toward a shared analytical backbone without changing technology.

• Compliance datastore — supporting data take-out (export), deletion, and anonymization. By centralizing a queryable copy of data, VaireDB makes it easier to satisfy regulatory obligations such as subject-access exports, right-to-be-forgotten deletions, and anonymization, without hunting through every individual service store.

Architecture

VaireDB follows a coordinator/worker topology:

┌──────────────────────────────────────────────────┐
│                   Clients                         │
│            (psql, JDBC, any PG driver)           │
└──────────────────────┬───────────────────────────┘
                       │ PostgreSQL wire protocol (port 5432)
                       ▼
┌──────────────────────────────────────────────────┐
│               Coordinator Node                    │
│  ┌─────────┐ ┌────────┐ ┌──────────────────┐    │
│  │ Catalog │ │ Router │ │ Ballista Sched.  │    │
│  └─────────┘ └────────┘ └──────────────────┘    │
└──────┬───────────────┬───────────────────────────┘
       │ gRPC writes   │ reads (Ballista)
       ▼               ▼
┌────────────┐  ┌────────────┐  ┌────────────┐
│ Core Node  │  │ Core Node  │  │ Core Node  │
│  (DuckDB)  │  │  (DuckDB)  │  │  (DuckDB)  │
└────────────┘  └────────────┘  └────────────┘

• Coordinator handles client connections, SQL parsing, query planning, metadata management, and dispatching work to core nodes.
• Core Nodes store data in DuckDB, execute shard-local queries, and replicate writes.

For full details, see the Architecture Documentation.

Project Documentation

VaireDB Documentation

System Design Documentation

All documentation lives under docs/, split into architecture references, feature designs, and testing notes.

Architecture (docs/specs/):

Features (docs/features/):

Getting Started

Prerequisites

• Rust 1.95 or later (2024 edition)
• Protobuf compiler (protoc) for gRPC code generation
• Make for build automation

Building

# Debug build
make build

# Release build
make build-release

# Type-check only (faster feedback loop)
make check

Running Locally - Binary

Start the coordinator:

make run-coordinator

In a separate terminal, start a core node:

make run-core

Connect with any PostgreSQL client:

psql -h localhost -p 5432

Running Locally - Docker Compose

Start a small VaireDB cluster (1 coordinator + 5 core node):

make e2e-up

Stop the small VaireDB cluster:

make e2e-down

Configuration

Configuration uses YAML files with environment-based overlays:

config/
├── coordinator/
│   └── config.yml   # Default coordinator configurations
└── core/
    └── config.yml   # Default core configurations

Default ports:

Contributing

Project Layout

vairedb/
├── config/                    # YAML configuration files
├── crates/
│   ├── vairedb-coordinator/   # Coordinator node binary
│   ├── vairedb-core/          # Core node binary
│   └── vairedb-common/        # Shared protobuf code, config, scan plans
├── docker/                    # Dockers file
├── docs/                      # Architecture and testing documentation
├── proto/vairedb/v1/          # Protobuf service definitions
├── tests/e2e/                 # E2E tests
└── Makefile                   # Build automation

Development Workflow

# Format code
make fmt

# Run linter (clippy, fails on warnings)
make lint

# Run all tests
make test

# Run tests for a single crate
cargo test --package vairedb-coordinator

# Run a specific test
cargo test --package vairedb-coordinator -- test_name

# Run E2E tests
make e2e

# Generate code coverage
make coverage

Key Modules

Coordinator (crates/vairedb-coordinator/src/):

Core Node (crates/vairedb-core/src/):

Shared (crates/vairedb-common/src/):

Code that must be identical on both sides of the wire. A distributed stage crosses the wire naming its functions, and the executor resolves those names in its own registry, so every function whose PostgreSQL semantics differ from the engine default is implemented once here and registered on every node that plans or executes a read.

Protobuf Definitions

Service definitions live in proto/vairedb/v1/:

• node_service.proto -- Register, Heartbeat (bidirectional stream), ReportFailure
• write_service.proto -- ExecuteWrite (DML dispatch to core nodes)
• catalog.proto -- Metadata messages (TableMeta, ColumnDef, ShardMeta, NodeMeta, AnonymizationSecret)
• error.proto -- Shared VdbErrorCode enum (query, storage, cluster, catalog, internal codes)

Code is generated automatically by vairedb-common/build.rs during cargo build.

License

Apache License 2.0 -- see LICENSE