Marmot v2

What & Why?

Marmot v2 is a leaderless, distributed SQLite replication system built on a gossip-based protocol with distributed transactions and eventual consistency.

Key Features:

• Leaderless Architecture: No single point of failure - any node can accept writes
• MySQL Protocol Compatible: Connect with any MySQL client (DBeaver, MySQL Workbench, mysql CLI)
• WordPress Compatible: Full MySQL function support for running distributed WordPress
• Distributed Transactions: Percolator-style write intents with conflict detection
• Multi-Database Support: Create and manage multiple databases per cluster
• DDL Replication: Distributed schema changes with automatic idempotency and cluster-wide locking
• Replicated Bulk Load: LOAD DATA LOCAL INFILE with distributed commit semantics
• Production-Ready SQL Parser: Powered by rqlite/sql AST parser for MySQL→SQLite transpilation
• CDC-Based Replication: Row-level change data capture for consistent replication
• Built-in Vector Search: Local IVF/PQ vector indexes for RAG workloads, with live CRUD and exact rerank from the base table

Why Marmot?

The Problem with Traditional Replication

MySQL active-active requires careful setup of replication, conflict avoidance, and monitoring. Failover needs manual intervention. Split-brain scenarios demand operational expertise. This complexity doesn't scale to edge deployments.

Marmot's Approach

• Zero operational overhead: Automatic recovery from split-brain via eventual consistency + anti-entropy
• No leader election: Any node accepts writes, no failover coordination needed
• Direct SQLite access: Clients can read the local SQLite file directly for maximum performance
• Tunable consistency: Choose ONE/QUORUM/ALL per your latency vs durability needs

Why MySQL Protocol?

• Ecosystem compatibility - existing drivers, ORMs, GUI tools work out-of-box
• Battle-tested wire protocol implementations
• Run real applications like WordPress without modification

Ideal Use Cases

Marmot excels at read-heavy edge scenarios:

When to Consider Alternatives

• Strong serializability required → CockroachDB, Spanner
• Single-region high-throughput → PostgreSQL, MySQL directly
• Large datasets (>100GB) → Sharded solutions

Quick Start

# Start a single-node cluster
./marmot-v2

# Or run as daemon (background)
./marmot-v2 -daemon -pid-file=/tmp/marmot/marmot.pid

# Connect with MySQL client
mysql -h localhost -P 3306 -u root

# Or use DBeaver, MySQL Workbench, etc.

Testing Replication

# Test DDL and DML replication across a 2-node cluster
./scripts/test-ddl-replication.sh

# This script will:
# 1. Start a 2-node cluster
# 2. Create a table on node 1 and verify it replicates to node 2
# 3. Insert data on node 1 and verify it replicates to node 2
# 4. Update data on node 2 and verify it replicates to node 1
# 5. Delete data on node 1 and verify it replicates to node 2

# Manual cluster testing
./examples/start-seed.sh              # Start seed node (port 8081, mysql 3307)
./examples/join-cluster.sh 2 localhost:8081  # Join node 2 (port 8082, mysql 3308)
./examples/join-cluster.sh 3 localhost:8081  # Join node 3 (port 8083, mysql 3309)

# Connect to any node and run queries
mysql --protocol=TCP -h localhost -P 3307 -u root
mysql --protocol=TCP -h localhost -P 3308 -u root

# Cleanup
pkill -f marmot-v2

WordPress Support

Marmot can run distributed WordPress with full database replication across nodes. Each WordPress instance connects to its local Marmot node, and all database changes replicate automatically.

MySQL Compatibility for WordPress

Marmot implements MySQL functions required by WordPress:

Quick Start: 3-Node WordPress Cluster

cd examples/wordpress-cluster
./run.sh up

This starts:

• 3 Marmot nodes with QUORUM write consistency
• 3 WordPress instances each connected to its local Marmot node

┌─────────────┐  ┌─────────────┐  ┌─────────────┐
│ WordPress-1 │  │ WordPress-2 │  │ WordPress-3 │
│ :9101       │  │ :9102       │  │ :9103       │
└──────┬──────┘  └──────┬──────┘  └──────┬──────┘
       ▼                ▼                ▼
┌─────────────┐  ┌─────────────┐  ┌─────────────┐
│  Marmot-1   │◄─┤  Marmot-2   │◄─┤  Marmot-3   │
│ MySQL: 9191 │  │ MySQL: 9192 │  │ MySQL: 9193 │
└─────────────┘  └─────────────┘  └─────────────┘
       └──────────────┴──────────────┘
              QUORUM Replication

Test it:

1. Open http://localhost:9101 - complete WordPress installation
2. Open http://localhost:9102 or http://localhost:9103
3. See your content replicated across all nodes!

Commands:

./run.sh status   # Check cluster health
./run.sh logs-m   # Marmot logs only
./run.sh logs-wp  # WordPress logs only
./run.sh down     # Stop cluster

Production Considerations for WordPress

• Media uploads: Use S3/NFS for shared media storage (files not replicated by Marmot)
• Sessions: Use Redis or database sessions for sticky-session-free load balancing
• Caching: Each node can use local object cache (Redis/Memcached per region)

Architecture

Marmot v2 uses a fundamentally different architecture from other SQLite replication solutions:

vs. rqlite/dqlite/LiteFS:

• ❌ They require a primary node for all writes
• ✅ Marmot allows writes on any node
• ❌ They use leader election (Raft)
• ✅ Marmot uses gossip protocol (no leader)
• ❌ They require proxy layer or page-level interception
• ✅ Marmot uses MySQL protocol for direct database access

How It Works:

1. Write Coordination: 2PC (Two-Phase Commit) with configurable consistency (ONE, QUORUM, ALL)
2. Conflict Resolution: Per-row Last-Write-Wins (LWW) on the transaction's commit HLC timestamp
3. Cluster Membership: SWIM-style gossip with failure detection
4. Data Replication: Full database replication - all nodes receive all data
5. DDL Replication: Cluster-wide schema changes with automatic idempotency

Comparison with Alternatives

DDL Replication

Marmot v2 supports distributed DDL (Data Definition Language) replication without requiring master election:

How It Works

1. Cluster-Wide Locking: Each DDL operation acquires a distributed lock per database (default: 30-second lease)

   • Prevents concurrent schema changes on the same database
   • Locks automatically expire if a node crashes
   • Different databases can have concurrent DDL operations

2. Automatic Idempotency: DDL statements are automatically rewritten for safe replay

   CREATE TABLE users (id INT)
   → CREATE TABLE IF NOT EXISTS users (id INT)
   
   DROP TABLE users
   → DROP TABLE IF EXISTS users
   

3. Schema Version Tracking: Each database maintains a schema version counter (__marmot_schema_version)

   • Incremented on every DDL operation
   • Exchanged via gossip protocol for drift detection
   • A joining node is not promoted until its schema versions match or exceed its alive peers'; anti-entropy restores a snapshot from the peer with the highest schema version

4. Rolling Upgrades: DDL and CREATE/DROP DATABASE return the retryable error 1213 while any cluster member does not serve the commit-log pull protocol (an older release). Upgrade every node, then retry.

5. Quorum-Based Replication: DDL replicates like DML through the same 2PC mechanism

   • No special master node needed
   • Works with existing consistency levels (QUORUM, ALL, etc.)

Configuration

[ddl]
# DDL lock lease duration (seconds)
lock_lease_seconds = 30

# Automatically rewrite DDL for idempotency
enable_idempotent = true

Best Practices

• ✅ Do: Execute DDL from a single connection/node at a time
• ✅ Do: Use qualified table names (mydb.users instead of users)
• ⚠️ Caution: ALTER TABLE is less idempotent - avoid replaying failed ALTER operations
• ❌ Don't: Run concurrent DDL on the same database from multiple nodes

CDC-Based Replication

Marmot v2 uses Change Data Capture (CDC) for replication instead of SQL statement replay:

How It Works

1. Row-Level Capture: Instead of replicating SQL statements, Marmot captures the actual row data changes (INSERT/UPDATE/DELETE)
2. Canonical Binary Row Format: DML changes are encoded once as msgpack EncodedCapturedRow records and stored in Marmot's CDC segment log
3. Minimal Wire Payload: DML replication sends the encoded row blob plus statement metadata; decoded before/after maps are local apply-time state only
4. Deterministic Application: Row data is applied directly to the target database, avoiding parsing ambiguities

Benefits

• Consistency: Same row data applied everywhere, no SQL parsing differences
• Performance: Binary format is more efficient than SQL text
• Reliability: No issues with SQL syntax variations between MySQL and SQLite
• Lower Write Amplification: Row payloads are not duplicated into Pebble intent keys; Pebble tracks transaction metadata and row locks while the CDC segment log stores DML bytes
• Atomic CDC Publication: DML bytes become visible through /cdc/manifest/{txnID} only after their segment ranges are known; recovery validates and truncates segment tails at record boundaries

Row Key Extraction

For UPDATE and DELETE operations, Marmot automatically extracts row keys:

• Uses PRIMARY KEY columns when available
• Falls back to ROWID for tables without explicit primary key
• Handles composite primary keys correctly

CDC Publisher

Marmot can publish CDC events to external messaging systems, enabling real-time data pipelines, analytics, and event-driven architectures. Events follow the Debezium specification for maximum compatibility with existing CDC tooling.

Features

• Debezium-Compatible Format: Events conform to the Debezium event structure, compatible with Kafka Connect, Flink, Spark, and other CDC consumers
• Multi-Sink Support: Publish to multiple destinations simultaneously (Kafka, NATS)
• Glob-Based Filtering: Filter which tables and databases to publish
• Automatic Retry: Exponential backoff with configurable limits
• Persistent Cursors: Survives restarts without losing position

Configuration

[publisher]
enabled = true

[[publisher.sinks]]
name = "kafka-main"
type = "kafka"                    # "kafka" or "nats"
format = "debezium"               # Debezium-compatible JSON format
brokers = ["localhost:9092"]      # Kafka broker addresses
topic_prefix = "marmot.cdc"       # Topics: {prefix}.{database}.{table}
filter_tables = ["*"]             # Glob patterns (e.g., "users", "order_*")
filter_databases = ["*"]          # Glob patterns (e.g., "prod_*")
batch_size = 100                  # Events per poll cycle
poll_interval_ms = 10             # Polling interval

# NATS sink example
[[publisher.sinks]]
name = "nats-events"
type = "nats"
format = "debezium"
nats_url = "nats://localhost:4222"
topic_prefix = "marmot.cdc"
filter_tables = ["*"]
filter_databases = ["*"]

Event Format

Events follow the Debezium envelope structure:

{
  "schema": { ... },
  "payload": {
    "before": null,
    "after": {"id": 1, "name": "alice", "email": "alice@example.com"},
    "source": {
      "version": "2.9.0-beta",
      "connector": "marmot",
      "name": "marmot",
      "ts_ms": 1702500000000,
      "db": "myapp",
      "table": "users"
    },
    "op": "c",
    "ts_ms": 1702500000000
  }
}

Operation Types (per Debezium spec):

Topic Naming

Topics follow the pattern: {topic_prefix}.{database}.{table}

Examples:

• marmot.cdc.myapp.users
• marmot.cdc.myapp.orders
• marmot.cdc.analytics.events

Use Cases

• Real-Time Analytics: Stream changes to data warehouses (Snowflake, BigQuery, ClickHouse)
• Event-Driven Microservices: Trigger actions on data changes
• Cache Invalidation: Keep caches in sync with database changes
• Audit Logging: Capture all changes for compliance
• Search Indexing: Keep Elasticsearch/Algolia in sync

For more details, see the Integrations documentation.

Edge Deployment Patterns

Lambda Sidecar

Deploy Marmot as a lightweight regional replica alongside Lambda functions:

• Local SQLite reads (sub-ms latency)
• Writes replicate to cluster
• No cold-start database connections

Read-Only Regional Replicas

Scale reads globally with replica mode and transparent failover:

[replica]
enabled = true
follow_addresses = ["central-cluster-1:8080", "central-cluster-2:8080", "central-cluster-3:8080"]
replicate_databases = []                    # Filter databases (empty = all, supports glob patterns)
database_discovery_interval_seconds = 10    # Poll for new databases (default: 10)
discovery_interval_seconds = 30             # Poll for cluster membership (default: 30)
failover_timeout_seconds = 60               # Failover timeout (default: 60)
snapshot_concurrency = 3                    # Parallel snapshot downloads (default: 3)
snapshot_cache_ttl_seconds = 30             # Snapshot cache TTL (default: 30)

• Discovers cluster nodes automatically via GetClusterNodes RPC
• Transparent failover when current source becomes unavailable
• Automatic discovery of new databases with configurable polling
• Per-database selective replication with glob pattern support
• Parallel snapshot downloads with configurable concurrency
• Snapshot caching for performance optimization
• Zero cluster participation overhead
• Auto-reconnect with exponential backoff

Hybrid: Edge Reads, Central Writes

• Deploy full cluster in central region
• Deploy read replicas at edge locations
• Application routes writes to central, reads to local replica

SQL Statement Compatibility

Marmot supports a wide range of MySQL/SQLite statements through its MySQL protocol server. The following table shows compatibility for different statement types:

Legend

• ✅ Full: Fully supported and working
• ✅ Parsed: Statement is parsed and recognized
• ⚠️ Limited: Works but has limitations in distributed context
• ❌ No: Not supported or not replicated
• N/A: Not applicable (read-only or session-local)

Important Notes

1. Schema Changes (DDL): DDL statements are fully replicated with cluster-wide locking and automatic idempotency. See the DDL Replication section for details.

2. XA Transactions: Marmot has its own distributed transaction protocol based on 2PC. MySQL XA transactions are not compatible with Marmot's replication model.

3. User Management (DCL): User and privilege management statements are local to each node. For production deployments, consider handling authentication at the application or proxy level.

4. Table Locking: LOCK TABLES statements are recognized but not enforced across the cluster. Use application-level coordination for distributed locking needs.

5. Qualified Names: Marmot fully supports qualified table names (e.g., db.table) in DML and DDL operations.

6. LOAD DATA Scope: Marmot supports LOAD DATA LOCAL INFILE (client-uploaded file bytes). LOAD DATA INFILE (server-side file access) is not supported.

Vector Search

Marmot has built-in vector similarity search for RAG-style workloads. No external vector service is required: embeddings stay in your table, DML replicates through Marmot CDC, and each node builds its own local ANN files for reads.

The SQL surface is intentionally small:

• CREATE VECTOR INDEX ... ON table(column) DIM n METRIC ...
• vec_match(column, query_vector, k) in WHERE
• vec_distance(column, query_vector) in ORDER BY
• REINDEX VECTOR index_name
• DROP VECTOR INDEX index_name

RAG Table Example

CREATE DATABASE rag;
USE rag;

CREATE TABLE chunks (
    id          INTEGER PRIMARY KEY,
    tenant_id   INTEGER NOT NULL,
    source_uri  TEXT NOT NULL,
    chunk_no    INTEGER NOT NULL,
    title       TEXT NOT NULL,
    body        TEXT NOT NULL,
    status      TEXT NOT NULL,
    created_at  INTEGER NOT NULL,
    embedding   BLOB NOT NULL
);

CREATE INDEX chunks_tenant_status_idx
    ON chunks(tenant_id, status);

CREATE VECTOR INDEX chunks_embedding_idx
    ON chunks(embedding)
    DIM 1536
    METRIC cosine;

The embedding column is a packed little-endian float32 blob. Use driver parameters rather than hand-writing blobs in SQL.

import struct

def vec_blob(values: list[float]) -> bytes:
    return struct.pack("<" + "f" * len(values), *values)

# Use your driver's placeholder convention; the important part is binding
# vec_blob(embedding) as bytes for the BLOB column.
cursor.execute(
    """
    INSERT INTO chunks
        (id, tenant_id, source_uri, chunk_no, title, body, status, created_at, embedding)
    VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
    """,
    (1, 42, "s3://kb/intro.md", 0, "Intro", "Marmot stores vectors in SQLite.", "published", 1730000000, vec_blob(embedding)),
)

Search composes with normal SQL filters:

SELECT id, source_uri, chunk_no, title, body
  FROM chunks
 WHERE vec_match(embedding, ?, 10)
   AND tenant_id = 42
   AND status = 'published'
 ORDER BY vec_distance(embedding, ?)
 LIMIT 10;

Pass the same query-vector blob to both placeholders.

LIMIT is the final row count. The third argument to vec_match is the ANN candidate and exact-rerank budget, so it may be larger than LIMIT when you want more recall headroom or selective post-filters:

SELECT id, title
  FROM chunks
 WHERE vec_match(embedding, ?, 100)
   AND tenant_id = 42
 ORDER BY vec_distance(embedding, ?)
 LIMIT 10;

Marmot rejects vec_match(..., k) when k < LIMIT.

Tuning Basics

For most RAG stores, start with auto tuning:

CREATE VECTOR INDEX chunks_embedding_idx
    ON chunks(embedding)
    DIM 1536
    METRIC cosine;

Override only when you have measured a reason:

CREATE VECTOR INDEX chunks_embedding_idx
    ON chunks(embedding)
    DIM 1536
    METRIC cosine
    WITH (nlist = 512, nprobe = 32);

SET @@marmot_vec_nprobe = 64;          -- per-connection recall/latency test
SET @@marmot_vec_force_plan = 'post';  -- force IVF path for debugging
SET @@marmot_vec_force_plan = 'auto';

nlist controls the number of IVF clusters. More clusters usually reduce bytes scanned per query but make build and maintenance work heavier. nprobe controls how many clusters a query scans. Higher nprobe usually improves recall and lowers false negatives, but increases disk reads and CPU per query. The default auto policy targets compact clusters and derives a probe budget from expected scanned rows.

Insert Visibility And Settling

Vector indexes are live:

• Immediately after insert/update/delete: committed rows are query-visible through the local overlay.
• First clustered publish: for empty-table create, Marmot publishes the first .vecseg generation from a bounded overlay snapshot once enough rows exist.
• Settled: background maintenance has folded the overlay into stable clusters, grown auto-tuned cluster count when needed, and reduced cluster skew.
• Manual rebuild: REINDEX VECTOR chunks_embedding_idx forces a full local retrain/rewrite.

Settling improves read latency and QPS; it is not required for correctness. Raw insert QPS and final query-ready QPS are different measurements.

Latest local 100K vector validation:

The same 100K force-build validation measured DDL create at 203ms, insert at 100,000 rows in 3.00s, first clustered publish at 15.886s, and final settled state at 1m08.286s.

Latest local 500K read validation:

The matching 500K force-build lifecycle run measured 500,000 inserts in 36.118s, first clustered publish at 1m21.011s, final settled state at 5m31.928s, and RSS after settled cleanup at 917 MB in the same process. Reopened steady read memory is lower because the transient build buffers are gone.

Runtime Model

• Base table: exact vectors and metadata remain in your user table.
• Local derived files: each node stores immutable .vecseg generations, rowmaps, block metadata sidecars, manifests, and an overlay journal next to the SQLite database.
• Approximate scan: high-dimensional stable rows use compact residual PQ8; low-dimensional stable rows use residual int8. Raw float32 stable segment payloads are not written.
• Overlay: rows are query-visible immediately. Before first publish the overlay may hold temporary prepared float32 bytes; after probe centroids exist, overlay rows are compact residual int8 refs and applied prefixes are compacted away.
• Exact rerank: Marmot fetches shortlisted exact vectors from the base table, materializes them in Go, and reranks every non-empty ANN candidate set before returning rows. Filtered Go-rank queries can widen candidate budgets to refill the final LIMIT.
• Replication: DML replicates as row-level CDC; vector index DDL replicates as compact control metadata. Segment files, rowmaps, block metadata sidecars, centroids, PQ codebooks, and overlay journals are local derived state, not replicated artifacts.

Metrics: l2, cosine, and dot. For dot, set WITH (max_norm = ...) to a fixed upper bound for vector norms.

Full startup guide, tuning guidance, lifecycle details, benchmarking flags, crash semantics, and limitations are in the Vector Search docs.

SQLite Extensions

Marmot also supports loading arbitrary SQLite extensions for anything that isn't vector search — full-text search tokenizers, JSON extensions, custom aggregates, domain-specific functions.

Configuration

[extensions]
# Directory containing extension libraries (.so, .dylib, .dll)
directory = "/opt/sqlite/extensions"

# Extensions loaded automatically into every connection
always_loaded = ["my-custom-ext"]

Dynamic Loading

Extensions can be loaded at runtime via SQL:

-- Load an extension by name (resolved from extensions.directory)
LOAD EXTENSION my-custom-ext;

Extension Resolution:

• Searches the configured directory for the extension
• Tries multiple patterns: name, name.so, name.dylib, libname.so, libname.dylib
• Platform-appropriate suffix is preferred (.dylib on macOS, .so on Linux, .dll on Windows)

Security:

• Extension names cannot contain path separators (no path traversal)
• Resolved paths must stay within the configured directory
• Only extensions in the configured directory can be loaded

Typical Uses

For vector search, use the native functions described above — no extension required.

Session-Level Transpilation Toggle

Marmot normally transpiles MySQL syntax to SQLite. You can disable this per-session to send raw SQLite SQL directly.

Usage

-- Disable MySQL→SQLite transpilation (raw SQLite mode)
SET marmot_transpilation = OFF;

-- Now you can use SQLite-specific syntax
SELECT sqlite_version();
PRAGMA table_info(users);

-- Re-enable transpilation
SET marmot_transpilation = ON;

-- Back to MySQL mode
SELECT NOW();  -- Transpiled to SQLite's datetime functions

When to Use

• Raw SQLite queries: Use SQLite-specific syntax like PRAGMA, ATTACH, etc.
• Performance testing: Compare transpiled vs raw query performance
• Debugging: See exactly what SQLite receives
• Advanced SQLite features: Access features not available through MySQL syntax

Note: Transpilation is enabled by default. The setting is per-session and not persisted.

MySQL Protocol & Metadata Queries

Marmot includes a MySQL-compatible protocol server, allowing you to connect using any MySQL client (DBeaver, MySQL Workbench, mysql CLI, etc.). The server supports:

Metadata Query Support

Marmot provides full support for MySQL metadata queries, enabling GUI tools like DBeaver to browse databases, tables, and columns:

• SHOW Commands: SHOW DATABASES, SHOW TABLES, SHOW COLUMNS FROM table, SHOW CREATE TABLE, SHOW INDEXES
• INFORMATION_SCHEMA: Queries against INFORMATION_SCHEMA.TABLES, INFORMATION_SCHEMA.COLUMNS, INFORMATION_SCHEMA.SCHEMATA, and INFORMATION_SCHEMA.STATISTICS
• Type Conversion: Automatic SQLite-to-MySQL type mapping for compatibility

These metadata queries are powered by the rqlite/sql AST parser, providing production-grade MySQL query compatibility.

Connecting with MySQL Clients

# Using mysql CLI
mysql -h localhost -P 3306 -u root

# Connection string for applications
mysql://root@localhost:3306/marmot

Recovery Scenarios

Marmot handles various failure and recovery scenarios automatically:

Network Partition (Split-Brain)

How it works:

1. During partition, only the majority side can commit writes (quorum enforcement)
2. When partition heals, each node pulls the commit logs of its alive peers (see Anti-Entropy below)
3. Each row keeps the version of its last change; an older change never overwrites a newer one (per-row LWW)
4. For equal HLC timestamps, the higher node ID wins

Node Failure & Recovery

A node's membership is persisted. A restarted node is not a quorum by itself, and a gracefully stopped node rejoins on restart. A node marked REMOVED stays out until an operator allows it back (see Cluster Membership Management).

Anti-Entropy Background Process:

Every anti_entropy_interval_seconds (default 30), each node pulls every alive peer's local commit log, for every database:

1. Durable cursors: Each (peer, database) pair has its own cursor, persisted after every page, so a restart resumes where it stopped. The node lists the peer's committed log entries after the cursor (ListCommittedLog RPC) and fetches the transactions it lacks (FetchTransactions RPC).
2. Exactly once: A replayed transaction writes an applied marker (__marmot_applied_txn) in the same SQLite transaction, so a transaction received from several peers applies once.
3. Unfinished local transactions: A transaction left prepared locally that a peer committed is committed locally with the peer's commit timestamp. An abandoned local begin is discarded and the transaction replayed. A prepare whose rows were lost is discarded and replayed from the peer.
4. Snapshot fallback: If a peer has truncated its log past this node's cursor, the node restores the database from a snapshot, re-applies its own local log, and resumes pulling.
5. Promotion: A joining node turns ALIVE only once every database has caught up with its alive peers.

Log Garbage Collection:

• A log entry is deleted once every current member has consumed it and it is older than gc_min_retention_hours
• An entry older than gc_max_retention_hours is deleted regardless; a member that was down longer restores by snapshot
• A departed member's consumed position is dropped, so it no longer holds back GC
• DELETE tombstones in the row-version table are purged in chunks, once they are older than the member count times gc_max_retention_hours, and never when gc_max_retention_hours is 0

Commit Timestamps and Per-Row Versions:

• The coordinator merges the participants' PREPARE clocks and picks one commit HLC timestamp, sent on COMMIT and stored in the log. Every node commits with it and merges it into its clock; on startup the clock is seeded past the highest recorded commit.
• Every applied row change stamps __marmot_row_version (table, key, timestamp, node, tombstone flag) in the same SQLite transaction. An incoming change applies only if its version is at least the stored one.
• A DELETE leaves a tombstone, so an older INSERT or UPDATE cannot bring the row back. A primary-key change stamps both the old and the new key. LOAD DATA stamps every row.
• DDL keeps versions consistent: dropping a table removes its versions, renaming re-keys them, and changing a table's primary key clears them.

Consistency Guarantees

Conflict Resolution:

• Concurrent writes to the same row conflict at PREPARE (write intents); only one commits
• Replicated and replayed changes resolve per row via LWW on the commit HLC timestamp, so every node converges on the same row regardless of the order changes arrive
• Tie-breaker: higher node ID wins for equal timestamps

Limitations

• Selective Table Watching: All tables in a database are replicated. Selective table replication is not supported.
• WAL Mode Required: SQLite must use WAL mode for reliable multi-process changes.
• Eventually Consistent: Rows may sync out of order. SERIALIZABLE transaction assumptions may not hold across nodes.
• Concurrent DDL: Avoid running concurrent DDL operations on the same database from multiple nodes (protected by cluster-wide lock with 30s lease).

Auto-Increment & ID Generation

The Problem with Distributed IDs

Distributed databases need globally unique IDs, but traditional solutions cause problems:

The JavaScript Problem:

// 64-bit ID from TiDB or other distributed DBs
const id = 7318624812345678901;
console.log(id);  // 7318624812345679000 - WRONG! Precision lost!

// JSON parsing also breaks
JSON.parse('{"id": 7318624812345678901}');  // {id: 7318624812345679000}

TiDB's answer? "Use strings." But that breaks ORMs, existing application code, and type safety.

Marmot's Solution: Compact ID Mode

For BIGINT AUTO_INCREMENT columns, Marmot offers two ID generation modes to solve this (narrower columns are covered in Narrow AUTO_INCREMENT Columns):

[mysql]
auto_id_mode = "compact"  # Safe for JavaScript (default)
# auto_id_mode = "extended"  # Full 64-bit for new systems

Compact Mode Guarantees:

• IDs stay under Number.MAX_SAFE_INTEGER (9,007,199,254,740,991)
• Still globally unique across all nodes
• Still monotonically increasing (per node)
• No silent precision loss in JSON/JavaScript
• Works with existing ORMs expecting integer IDs

With Marmot compact mode:

const id = 4503599627370496;
console.log(id);  // 4503599627370496 - Correct!
JSON.parse('{"id": 4503599627370496}');  // {id: 4503599627370496} - Correct!

How Auto-Increment Works

1. DDL Transformation: An integer AUTO_INCREMENT column is stored as SQLite INTEGER. A column declared narrower than BIGINT keeps its declared width (and UNSIGNED) in a comment marker in the stored schema.

   CREATE TABLE users (id BIGINT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(100))
   -- Becomes internally:
   CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)
   

2. DML ID Injection: When inserting with 0 or NULL for a BIGINT auto-increment column:

   INSERT INTO users (id, name) VALUES (0, 'alice')
   -- Becomes internally (compact mode):
   INSERT INTO users (id, name) VALUES (4503599627370496, 'alice')
   

3. Explicit IDs Preserved: If you provide an explicit non-zero ID in a BIGINT column, it is used as-is.

Schema-Based Detection:

Marmot automatically detects auto-increment columns by querying SQLite schema directly:

• Single-column INTEGER PRIMARY KEY (SQLite rowid alias), with the declared width read from its marker
• Single-column BIGINT PRIMARY KEY (Marmot's transformed columns)

No registration required - columns are detected from schema at runtime, works across restarts, and works with existing databases.

Narrow AUTO_INCREMENT Columns

TINYINT, SMALLINT, MEDIUMINT and INT AUTO_INCREMENT columns get ids that fit their declared width. Each node takes ids from ranges of the column it claims through a quorum two-phase commit, so no two nodes issue the same id while cluster membership is stable.

• On a node that is still merging claim bases from its peers after startup, an insert waits up to transaction.lock_wait_timeout_seconds and then returns the retryable 1205; inside a transaction that has already written a row, it returns 1205 at once.
• An explicit id outside the declared width returns 1264; an id another node may issue returns 1235; a column whose ids are used up returns 1062.
• Ids are not reset by DROP TABLE/CREATE TABLE or DROP DATABASE/CREATE DATABASE.

The full rules are in Compatibility; adding or removing nodes follows the procedure in Operations.

Configuration

Marmot v2 uses a TOML configuration file (default: config.toml). All settings have sensible defaults.

Core Configuration

node_id = 0  # 0 = auto-generate
data_dir = "./marmot-data"

Transaction Manager

[transaction]
heartbeat_timeout_seconds = 10  # Transaction timeout without heartbeat
conflict_window_seconds = 10    # Conflict resolution window
lock_wait_timeout_seconds = 50  # Lock wait timeout (MySQL: innodb_lock_wait_timeout)

An insert into a narrow AUTO_INCREMENT column also waits up to lock_wait_timeout_seconds while its node is still merging claim bases after startup (see Narrow AUTO_INCREMENT Columns).

Note: Transaction log garbage collection is managed by the replication configuration to coordinate with anti-entropy. See replication.gc_min_retention_hours and replication.gc_max_retention_hours.

Connection Pool

[connection_pool]
pool_size = 4              # Number of SQLite connections
max_idle_time_seconds = 10 # Max idle time before closing
max_lifetime_seconds = 300 # Max connection lifetime (0 = unlimited)

gRPC Client

[grpc_client]
keepalive_time_seconds = 10    # Keepalive ping interval
keepalive_timeout_seconds = 3  # Keepalive ping timeout
max_retries = 3                # Max retry attempts
retry_backoff_ms = 100         # Retry backoff duration

Coordinator

[coordinator]
prepare_timeout_ms = 2000 # Prepare phase timeout
commit_timeout_ms = 2000  # Commit phase timeout
abort_timeout_ms = 2000   # Abort phase timeout

Cluster

[cluster]
grpc_bind_address = "0.0.0.0"
grpc_port = 8080
seed_nodes = []                # List of seed node addresses
cluster_secret = ""            # PSK for cluster authentication (see Security section)
gossip_interval_ms = 1000      # Gossip interval
gossip_fanout = 3              # Number of peers to gossip to
suspect_timeout_ms = 5000      # Suspect timeout
dead_timeout_ms = 10000        # Dead timeout
autoinc_merge_interval_ms = 2000       # Narrow AUTO_INCREMENT: retry merging claim bases while votes are held
autoinc_base_sync_interval_ms = 10000  # Narrow AUTO_INCREMENT: raise claim bases to every alive peer's

Security

Marmot supports Pre-Shared Key (PSK) authentication for cluster communication. This is strongly recommended for production deployments.

[cluster]
# All nodes in the cluster must use the same secret
cluster_secret = "your-secret-key-here"

Environment Variable (Recommended):

For production, use the environment variable to avoid storing secrets in config files:

export MARMOT_CLUSTER_SECRET="your-secret-key-here"
./marmot

The environment variable takes precedence over the config file.

Generating a Secret:

# Generate a secure random secret
openssl rand -base64 32

Behavior:

• If cluster_secret is empty and MARMOT_CLUSTER_SECRET is not set, authentication is disabled
• A warning is logged at startup when authentication is disabled
• All gRPC endpoints (gossip, replication, snapshots) are protected when authentication is enabled
• Nodes with mismatched secrets will fail to communicate (connection rejected with "invalid cluster secret")

Cluster Membership Management

Marmot provides admin HTTP endpoints for managing cluster membership (requires cluster_secret to be configured):

Node Lifecycle:

• New/restarted nodes auto-join via gossip - no manual intervention needed
• Nodes marked REMOVED via admin API cannot auto-rejoin - must be explicitly allowed
• This prevents decommissioned nodes from accidentally rejoining the cluster

# View cluster members and quorum info
curl -H "X-Marmot-Secret: your-secret" http://localhost:8080/admin/cluster/members

# Remove a node from the cluster (excludes from quorum, blocks auto-rejoin)
curl -X POST -H "X-Marmot-Secret: your-secret" http://localhost:8080/admin/cluster/remove/2

# Allow a removed node to rejoin (node must then restart to join)
curl -X POST -H "X-Marmot-Secret: your-secret" http://localhost:8080/admin/cluster/allow/2

See the Operations documentation for detailed usage and examples.

Replica Mode

For read-only replicas that follow cluster nodes with transparent failover:

[replica]
enabled = true                                   # Enable read-only replica mode
follow_addresses = ["node1:8080", "node2:8080", "node3:8080"]  # Seed nodes for discovery
secret = "replica-secret"                        # PSK for authentication (required)
replicate_databases = []                         # Filter databases (empty = all, supports glob patterns like "prod_*")
database_discovery_interval_seconds = 10         # Poll for new databases (default: 10)
discovery_interval_seconds = 30                  # Poll for cluster membership (default: 30)
failover_timeout_seconds = 60                    # Max time to find alive node during failover (default: 60)
reconnect_interval_seconds = 5                   # Reconnect delay on disconnect (default: 5)
reconnect_max_backoff_seconds = 30               # Max reconnect backoff (default: 30)
initial_sync_timeout_minutes = 30                # Timeout for initial snapshot (default: 30)
snapshot_concurrency = 3                         # Parallel snapshot downloads (default: 3)
snapshot_cache_ttl_seconds = 30                  # Snapshot cache TTL in seconds (default: 30)
forward_writes = false                           # Forward writes to cluster nodes (default: false)
forward_write_timeout_sec = 30                   # Timeout for forwarded writes (default: 30s)

You can also specify follow addresses via CLI:

./marmot --config=replica.toml --follow-addresses=node1:8080,node2:8080,node3:8080

Per-Database Selective Replication:

Control which databases are replicated using the replicate_databases filter:

[replica]
# Replicate only specific databases
replicate_databases = ["myapp", "analytics"]

# Replicate databases matching glob patterns
replicate_databases = ["prod_*", "staging_*"]

# Replicate all databases (default)
replicate_databases = []

The system database (__marmot_system) is never replicated - each replica maintains its own independent system database.

Snapshot Caching:

Replicas use an LRU cache to avoid redundant snapshot creation:

• Cache TTL controlled by snapshot_cache_ttl_seconds (default: 30 seconds)
• Cached snapshots served from temp storage until expiration
• Background cleanup runs automatically
• Improves performance when multiple replicas bootstrap simultaneously

Parallel Snapshot Downloads:

Control download concurrency with snapshot_concurrency:

• Downloads multiple database snapshots in parallel (default: 3)
• Uses worker pool pattern to limit resource usage
• Partial failure handling: continues even if some databases fail
• Failed databases retry in background with exponential backoff

Write forwarding in replica mode: Reads stay local. If forward_writes = true, writes (including LOAD DATA LOCAL INFILE) are transparently proxied to a writable cluster node.

Note: Replica mode is mutually exclusive with cluster mode. A replica receives all data via streaming replication and automatically discovers cluster nodes and fails over to another node if the current source becomes unavailable.

Replication

[replication]
default_write_consistency = "QUORUM"      # Write consistency level: ONE, QUORUM, ALL
default_read_consistency = "LOCAL_ONE"    # Read consistency level
write_timeout_ms = 5000                   # Write operation timeout
read_timeout_ms = 2000                    # Read operation timeout

# Anti-Entropy: each node pulls every alive peer's commit log (snapshot when a peer's log no longer covers it)
enable_anti_entropy = true                 # Enable the background log pull
anti_entropy_interval_seconds = 30         # How often each node pulls its peers' logs (default: 30s)
gc_interval_seconds = 60                   # GC interval (MUST be >= anti_entropy_interval)
delta_sync_threshold_transactions = 10000  # Read replicas only: snapshot instead of catch-up above this txn gap
delta_sync_threshold_seconds = 3600        # Only used by the GC retention checks below

# Garbage Collection: Reclaim disk space by deleting old transaction records
# - A log entry is deleted once every member consumed it and it is older than gc_min,
#   or once it is older than gc_max regardless
# - Set gc_max = 0 for unlimited retention
gc_min_retention_hours = 2
gc_max_retention_hours = 24

Anti-Entropy Tuning:

• Small clusters (2-3 nodes): Use default settings (30s AE, 60s GC)
• Large clusters (5+ nodes): Consider increasing AE interval to 60-120s and GC to 2x that value
• Long-running clusters: Keep gc_max_retention_hours at 24+ so a node down for a while catches up from the log instead of a snapshot

GC Configuration Rules (validated at startup when enable_anti_entropy is true):

• gc_interval_seconds must be >= anti_entropy_interval_seconds
• gc_min_retention_hours must be >= delta_sync_threshold_seconds (in hours)
• gc_max_retention_hours must be >= 2x delta_sync_threshold_seconds (in hours) and > gc_min_retention_hours, unless it is 0 (unlimited)
• Violating these rules will cause startup failure with helpful error messages

Query Pipeline

[query_pipeline]
transpiler_cache_size = 10000  # LRU cache for MySQL→SQLite transpilation
validator_pool_size = 8        # SQLite connection pool for validation

MySQL Protocol Server

[mysql]
enabled = true
bind_address = "0.0.0.0"
port = 3306
max_connections = 1000
unix_socket = ""              # Unix socket path (empty = disabled)
unix_socket_perm = 0660       # Socket file permissions
local_infile_enabled = true   # Enable LOAD DATA LOCAL INFILE
auto_id_mode = "compact"      # "compact" (53-bit, JS-safe) or "extended" (64-bit)

Unix Socket Connection (lower latency than TCP):

mysql --socket=/tmp/marmot/mysql.sock -u root

LOAD DATA LOCAL INFILE requirements:

• Server: mysql.local_infile_enabled = true
• Client capability enabled (for example: mysql --local-infile=1, or driver-specific local infile flag)

CDC Publisher

[publisher]
enabled = false  # Enable CDC publishing to external systems

[[publisher.sinks]]
name = "kafka-main"              # Unique sink name
type = "kafka"                   # "kafka" or "nats"
format = "debezium"              # Debezium-compatible JSON (only option)
brokers = ["localhost:9092"]     # Kafka broker addresses
topic_prefix = "marmot.cdc"      # Topic pattern: {prefix}.{db}.{table}
filter_tables = ["*"]            # Glob patterns for table filtering
filter_databases = ["*"]         # Glob patterns for database filtering
batch_size = 100                 # Events to read per poll cycle
poll_interval_ms = 10            # Polling interval (default: 10ms)
retry_initial_ms = 100           # Initial retry delay on failure
retry_max_ms = 30000             # Max retry delay (30 seconds)
retry_multiplier = 2.0           # Exponential backoff multiplier

See the Integrations documentation for details on event format, Kafka/NATS configuration, and use cases.

SQLite Extensions

[extensions]
directory = "/opt/sqlite/extensions"  # Search path for extensions
always_loaded = ["my-custom-ext"]     # Auto-load into every connection

Loading Extensions at Runtime:

LOAD EXTENSION my-custom-ext;

Vector search is built-in and does not require an extension. See the Vector Search section above.

Logging

[logging]
verbose = false          # Enable verbose logging
format = "json"          # Log format: "console" or "json" (json is faster)
file = ""                # Log file path (empty = stdout only)
max_size_mb = 100        # Max size in MB before rotation
max_backups = 5          # Number of old log files to keep
compress = true          # Compress rotated files with gzip

File Logging with Rotation:

When file is set, logs are written to both stdout and the specified file. The file is automatically rotated when it reaches max_size_mb, keeping the last max_backups files. Rotated files are optionally compressed with gzip.

[logging]
file = "/var/log/marmot/marmot.log"
max_size_mb = 100
max_backups = 5
compress = true

Prometheus Metrics

[prometheus]
enabled = true  # Metrics served on gRPC port at /metrics endpoint

Accessing Metrics:

# Metrics are multiplexed with gRPC on the same port
curl http://localhost:8080/metrics

# Prometheus scrape config
scrape_configs:
  - job_name: 'marmot'
    static_configs:
      - targets: ['node1:8080', 'node2:8080', 'node3:8080']

See config.toml for complete configuration reference with detailed comments.

Benchmarks

Latest measured Pika write benchmark on a local development machine (Apple M-series, 3-node cluster, single machine):

Test Configuration

Load Phase (INSERT-only)

Latency

Replication Verification

All 3 nodes ended with identical row counts: 500,000 rows per node. Sample verification checked 100 rows and all 100 matched.

Note: These benchmarks are from a local development machine with all nodes on the same host. Production deployments across multiple machines will have different characteristics based on network latency. Expect P99 latencies of 50-200ms for cross-region QUORUM writes.

Backup & Disaster Recovery

Option 1: Litestream (Recommended)

Marmot's SQLite files are standard WAL-mode databases, compatible with Litestream:

litestream replicate /path/to/marmot-data/*.db s3://bucket/backup

Option 2: CDC to External Storage

Enable CDC publisher to stream changes to Kafka/NATS, then archive to your preferred storage.

Option 3: Filesystem Snapshots

Since Marmot uses SQLite with WAL mode, you can safely snapshot the data directory during operation.

Stargazers over time

FAQs & Community

• For FAQs visit this page
• For community visit our discord or discussions on GitHub