ExESDB - A BEAM-native Event Store

ExESDB is a BEAM-native Event Store, built on top of the khepri and ra subsystems.

Motivation

One of the arguments for BEAM development is that it comes "batteries included". Be it caching, storage, pub/sub, observability etc... the Erlang ecosystem always has the option to avoid external dependencies.

For Event Sourcing use cases however, the Event Store is often a separate service.

This project is an attempt at addressing this point, by building further upon the work of the rabbitmq/khepri and rabbitmq/ra subsystems.

Features

ExESDB is a distributed, BEAM-native Event Store that provides high-availability event sourcing capabilities with automatic cluster formation and coordination. Built on top of Khepri and Ra (Raft consensus), it offers enterprise-grade reliability and performance.

Core Event Store Functionality

Event Stream Management

• Stream Creation: Automatic stream creation on first event append
• Event Appending: Atomic append operations with optimistic concurrency control
• Event Retrieval: Query events with forward/backward traversal support
• Stream Versioning: Track stream versions for conflict detection and resolution
• Stream Listing: Enumerate all streams in the store

Event Storage

• Persistent Storage: Durable event storage using Khepri's distributed key-value store
• ACID Compliance: Atomic, consistent, isolated, and durable operations
• Conflict Resolution: Built-in optimistic concurrency control
• Data Integrity: Checksum validation and corruption detection

Subscription System

• Multiple Subscription Types:
  • :by_stream - Subscribe to specific event streams
  • :by_event_type - Subscribe to events by type classification
  • :by_event_pattern - Pattern-based event matching
  • :by_event_payload - Content-based subscription filtering
• Persistent Subscriptions: Durable subscriptions that survive node restarts
• Transient Subscriptions: Temporary subscriptions for real-time processing
• Event Replay: Start subscriptions from any stream version
• Acknowledgment System: Reliable event delivery with ACK/NACK support
• "Follow-the-Leader": Subscription processes automatically migrate to cluster leader

Snapshot Management

• Aggregate Snapshots: Store and retrieve aggregate state snapshots
• Version-based Snapshots: Snapshots tied to specific stream versions
• Snapshot Lifecycle: Create, read, update, and delete snapshot operations
• Performance Optimization: Reduce replay time for large aggregates
• Distributed Storage: Snapshots stored across the cluster for availability

Distributed Architecture & Clustering

LibCluster Integration

ExESDB uses LibCluster for automatic cluster discovery and formation:

• Strategy: Gossip-based multicast discovery
• Protocol: UDP multicast on configurable port (default: 45892)
• Network: Automatic node discovery on shared networks
• Security: Shared secret authentication for cluster membership
• Broadcast Discovery: Configurable multicast addressing

Cluster Formation Process

1. Initialization: Node starts and initializes LibCluster topology
2. Discovery: Uses gossip multicast to discover peer nodes
3. Authentication: Validates cluster membership using shared secrets
4. Coordination: ClusterCoordinator manages join/leave operations
5. Consensus: Khepri cluster formation using Raft consensus
6. Monitoring: Continuous health monitoring and leader election

High Availability Features

• Automatic Clustering: Nodes automatically discover and join clusters
• Split-Brain Prevention: ClusterCoordinator prevents network partition issues
• Leader Election: Automatic leader election using Raft consensus
• Failover: Seamless handling of node failures
• Data Replication: Events replicated across cluster nodes
• Consensus Protocol: Ra/Raft ensures data consistency

Storage Engine

Khepri Integration

• Distributed Tree Store: Hierarchical key-value storage
• MVCC: Multi-version concurrency control
• Transactions: ACID transaction support
• Schema Evolution: Support for data structure changes
• Triggers: Event-driven data processing

Ra (Raft) Consensus

• Strong Consistency: Linearizable read/write operations
• Partition Tolerance: Operates correctly during network partitions
• Leader-based Replication: Single leader for write operations
• Log Compaction: Automatic cleanup of old log entries
• Snapshot Support: Efficient state transfer for new nodes

Configuration & Deployment

Environment Configuration

• EX_ESDB_STORE_ID: Unique identifier for the store instance
• EX_ESDB_DB_TYPE: Deployment type (:single or :cluster)
• EX_ESDB_DATA_DIR: Data directory for persistent storage
• EX_ESDB_TIMEOUT: Operation timeout configuration
• EX_ESDB_CLUSTER_SECRET: Shared secret for cluster authentication
• EX_ESDB_COOKIE: Erlang distribution cookie
• EX_ESDB_PUB_SUB: PubSub configuration for event broadcasting

LibCluster Configuration

config :libcluster,
  topologies: [
    ex_esdb_cluster: [
      strategy: Cluster.Strategy.Gossip,
      config: [
        port: 45_892,
        if_addr: "0.0.0.0",
        multicast_addr: "255.255.255.255",
        broadcast_only: true,
        secret: System.get_env("EX_ESDB_CLUSTER_SECRET")
      ]
    ]
  ]

Gateway API Integration

High-Availability Proxy

• Load Balancing: Distribute requests across gateway workers
• Service Discovery: Automatic discovery of available gateway workers
• Fault Tolerance: Handle worker failures gracefully
• Request Routing: Smart routing based on operation type

Worker Distribution

• Swarm Integration: Distributed worker management
• Process Migration: Workers can move between cluster nodes
• Resource Management: Efficient resource utilization across cluster
• Monitoring: Real-time worker health and performance tracking

Operational Features

Monitoring & Observability

• Cluster Status: Real-time cluster membership and health
• Leader Tracking: Monitor current cluster leader
• Performance Metrics: Operation latency and throughput
• Error Tracking: Comprehensive error logging and reporting
• Health Checks: Built-in health check endpoints

Development Tools

• Cluster Manager: Interactive cluster management script
• Docker Compose: Multi-node development environment
• Chaos Engineering: Built-in chaos testing capabilities
• Validation Scripts: Automated cluster validation tools

Network Topology

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   ExESDB Node   │    │   ExESDB Node   │    │   ExESDB Node   │
│    (Leader)     │◄──►│   (Follower)    │◄──►│   (Follower)    │
│   Khepri + Ra   │    │   Khepri + Ra   │    │   Khepri + Ra   │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         ▲                       ▲                       ▲
         │                       │                       │
         │     Gossip Multicast Network (UDP:45892)     │
         │                       │                       │
         ▼                       ▼                       ▼
    Raft Consensus           Event Storage           Subscription
     & Replication          & Retrieval             Management

Deployment Scenarios

Single Node Deployment

• Development: Local development and testing
• Small Applications: Simple event sourcing needs
• Embedded Usage: Integration within existing applications

Multi-Node Cluster

• Production: High-availability production deployments
• Horizontal Scaling: Scale read/write capacity
• Geographic Distribution: Multi-region deployments
• Fault Tolerance: Survive individual node failures

Container Orchestration

• Docker Compose: Development and testing environments
• Kubernetes: Production container orchestration
• Docker Swarm: Simplified container clustering
• Health Checks: Container-level health monitoring

Performance Characteristics

Throughput

• Write Performance: Optimized for high-volume event appending
• Read Performance: Efficient event retrieval and streaming
• Concurrent Operations: Handle multiple simultaneous operations
• Batch Processing: Support for batch event operations

Scalability

• Horizontal Scaling: Add nodes to increase capacity
• Storage Scalability: Distributed storage across cluster
• Subscription Scaling: Distribute subscription load
• Resource Utilization: Efficient use of available resources

Integration Capabilities

BEAM Ecosystem

• Phoenix Integration: Real-time web applications
• LiveView Support: Real-time UI updates
• GenServer Integration: Native BEAM process integration
• OTP Supervision: Fault-tolerant supervision trees

External Systems

• REST APIs: HTTP-based integration
• Message Queues: Integration with external queuing systems
• Databases: Projection and read model support
• Monitoring Systems: Metrics and alerting integration

Installation

Docker Installation

ExESDB is available as a Docker image on Docker Hub with automatic versioning based on the mix.exs version.

Available Tags

• beamcampus/ex_esdb:latest - Latest build from master branch
• beamcampus/ex_esdb:0.0.18 - Specific version (current version)
• beamcampus/ex_esdb:0.0.x - Any specific version tag

Quick Start

Single Node:

docker run -d \
  --name ex-esdb \
  -p 4369:4369 \
  -p 9000-9100:9000-9100 \
  -p 45892:45892/udp \
  -e EX_ESDB_STORE_ID="my-store" \
  -e EX_ESDB_DB_TYPE="single" \
  -e EX_ESDB_DATA_DIR="/data" \
  -v ex-esdb-data:/data \
  beamcampus/ex_esdb:latest

Multi-Node Cluster:

# Node 1 (seed node)
docker run -d \
  --name ex-esdb-node1 \
  --network ex-esdb-net \
  -p 4369:4369 \
  -p 9001:9000 \
  -p 45892:45892/udp \
  -e EX_ESDB_STORE_ID="cluster-store" \
  -e EX_ESDB_DB_TYPE="cluster" \
  -e EX_ESDB_DATA_DIR="/data" \
  -e EX_ESDB_CLUSTER_SECRET="your-secret-key" \
  -e EX_ESDB_COOKIE="your-erlang-cookie" \
  -v ex-esdb-node1-data:/data \
  beamcampus/ex_esdb:latest

# Node 2
docker run -d \
  --name ex-esdb-node2 \
  --network ex-esdb-net \
  -p 9002:9000 \
  -e EX_ESDB_STORE_ID="cluster-store" \
  -e EX_ESDB_DB_TYPE="cluster" \
  -e EX_ESDB_DATA_DIR="/data" \
  -e EX_ESDB_CLUSTER_SECRET="your-secret-key" \
  -e EX_ESDB_COOKIE="your-erlang-cookie" \
  -v ex-esdb-node2-data:/data \
  beamcampus/ex_esdb:latest

# Node 3
docker run -d \
  --name ex-esdb-node3 \
  --network ex-esdb-net \
  -p 9003:9000 \
  -e EX_ESDB_STORE_ID="cluster-store" \
  -e EX_ESDB_DB_TYPE="cluster" \
  -e EX_ESDB_DATA_DIR="/data" \
  -e EX_ESDB_CLUSTER_SECRET="your-secret-key" \
  -e EX_ESDB_COOKIE="your-erlang-cookie" \
  -v ex-esdb-node3-data:/data \
  beamcampus/ex_esdb:latest

Docker Compose

For development and testing, use the provided Docker Compose setup:

# Clone the repository
git clone https://github.com/beam-campus/ex-esdb.git
cd ex-esdb/dev-env

# Start a 3-node cluster
./start-core-only.sh

# Or use the interactive cluster manager
./ez-cluster.sh

The Docker Compose setup includes:

• Core Cluster: 3-node ExESDB cluster (ex-esdb0, ex-esdb1, ex-esdb2)
• Extended Tier: Additional 2 nodes (ex-esdb10, ex-esdb11)
• Massive Tier: Additional 8 nodes (ex-esdb20-27)
• Automatic Networking: Configured Docker networks for cluster communication
• Data Persistence: Named volumes for data persistence
• Health Checks: Built-in container health monitoring

Environment Variables

📌 Important: When using the dev-env Docker Compose configurations, you must export the EX_ESDB_COOKIE environment variable on your host machine. This single environment variable is used for all cluster authentication purposes (cookies, secrets, etc.).

export EX_ESDB_COOKIE="your-secure-cookie-value"

Ports

Health Checks

The Docker image includes a built-in health check script:

# Check container health
docker exec ex-esdb ./check-ex-esdb.sh

# View health status
docker inspect --format='{{.State.Health.Status}}' ex-esdb

Production Considerations

1. Security: Use strong, unique values for EX_ESDB_CLUSTER_SECRET and EX_ESDB_COOKIE
2. Networking: Ensure proper firewall rules for cluster communication
3. Storage: Use named volumes or bind mounts for data persistence
4. Monitoring: Implement external monitoring for cluster health
5. Backups: Regular backup of data volumes
6. Resource Limits: Set appropriate CPU and memory limits

Hex Installation

ExESDB is also available as a Hex package for direct integration:

def deps do
  [
    {:ex_esdb, "~> 0.0.18"}
  ]
end

Contents

• Getting Started

Releases

• On Hex
• Release Documentation