Awesome Object Storage Native 
Open-source databases, application runtimes, and storage engines that use object storage for durable state, logs, indexes, or transactional datasets.
Entries are open-source implementations with a documented configuration in which object storage, such as Amazon S3, an S3-compatible service, Google Cloud Storage, or Azure Blob Storage, holds authoritative state, retained records, indexes, or transactional datasets. Some commit directly to object storage; others acknowledge writes through a separate durable tier first. Object-storage native does not mean bucket-only infrastructure: many entries also need a catalog, coordinator, or metadata database. Backup-only tools and general file-query engines are listed under Related Tools.
The comparison records object-storage backends, licenses, the role object storage plays, dependencies, and caveats for each entry. Experimental implementations are kept in a separate list.
Contents
Stateful Applications and Durable Actors
- Celld - Self-hosted Workers applications and Durable Objects with per-object SQLite and bucket-held state.
- Rivet Actors - Stateful actors with durable SQLite, realtime connections, queues, and S3-tiered storage; forthcoming v3 moves latency-tolerant broadcasts to S3 logs.
- Terse Durable Actors - Stateful TypeScript and Python actors with state snapshots in Google Cloud Storage.
Streaming and Messaging
- AutoMQ - Kafka-compatible event streaming with shared logs on S3-compatible object storage.
- Nisshi - Kafka-compatible broker with selectable storage engines, including S3.
- S2 Lite - Self-hostable server for the S2 durable streams API, built on SlateDB.
- Ursa for Apache Kafka - Kafka distribution with opt-in diskless topics stored through Ursa.
Databases and Data Processing
- Apache Druid - Real-time analytical database with durable segments in object-backed deep storage.
- ClickHouse - Analytical database whose MergeTree tables can use S3 or Azure Blob Storage disks.
- Embucket - DataFusion query engine with partial Snowflake compatibility and Iceberg tables in Amazon S3 Tables.
- Neon - PostgreSQL with separated compute and storage, branching, and object-backed history.
- RisingWave - Streaming SQL database that keeps state, tables, and materialized views in object storage.
- XTDB - Bitemporal SQL database with object-backed storage and a separately configured transaction log.
Search and Retrieval
- Chroma - Search database whose distributed deployment uses object-backed indexes and the wal3 log.
- HelixDB - Graph database with vector and full-text search over SlateDB storage on S3-compatible buckets.
- LanceDB - Embedded vector and multimodal database over Lance datasets in S3, Google Cloud Storage, or Azure Blob Storage.
- Milvus - Vector database with collection data and indexes in object storage and a separately configured write-ahead log.
- OpenData Vector - Early vector database with metadata filtering on SlateDB storage.
Transactional Tables and Array Datasets
- Apache Hudi - Lakehouse tables with upserts, deletes, and incremental queries, keeping data and metadata in cloud storage.
- Apache Iceberg - Analytical table format with object-backed files and catalog-coordinated commits.
- Apache Paimon - Lakehouse tables for streaming updates, change data, and analytical queries.
- Delta Lake - Transactional analytical tables with an object-backed log, also implemented natively in Rust by delta-rs.
- DuckLake - Transactional Parquet datasets with data files in object storage and a SQL database as the catalog.
- Icechunk - Transactional Zarr array storage with snapshots, branches, and no external catalog.
Observability
- Grafana Loki - Log aggregation and LogQL queries over object-backed chunks and indexes.
- Grafana Mimir - Scalable Prometheus metrics with object-backed blocks and a separate ingestion tier.
- Grafana Tempo - Distributed tracing and TraceQL over object-backed trace blocks.
- GreptimeDB - Observability database for metrics, logs, and traces with object-backed columnar storage.
- InfluxDB 3 Core - Time series database that stores its write-ahead log and Parquet data in object storage.
- OpenObserve - Logs, metrics, and traces with Parquet data in object storage.
- Parseable - SQL log analytics with data stored in object storage.
- Quickwit - Search engine for logs and traces with indexes stored in object storage.
- Thanos - Global Prometheus queries and long-term metrics in object-backed TSDB blocks.
Files, Volumes, and Repositories
- JuiceFS - Shared POSIX filesystem with file data in object storage and metadata in a separate database.
- Walgit - Git and Git LFS server that keeps repositories in a bucket and commits pushes with manifest compare-and-swap.
- ZeroFS - Filesystem and block server over NFS, 9P, and NBD with data and metadata in object storage; durability differs by protocol.
Building Storage Systems
- Graft - Alpha transactional page storage over object storage, with a SQLite extension.
- SlateDB - Embedded LSM key-value store that writes its write-ahead log and sorted tables to object storage.
- Tonbo - Alpha embedded database with Parquet tables and manifest compare-and-swap on S3-compatible storage.
- Ursa - Embeddable stream storage engine with object-backed logs and optional Iceberg or Delta Lake materialization.
- wal3 - Chroma's Rust log library built from immutable object fragments and conditional manifest updates.
- Woodpecker - Write-ahead log library and service on object storage, with etcd for metadata and coordination.
- DuckDB - In-process analytical database that can read and write Parquet, CSV, and JSON files on object storage.
- Litestream - Asynchronous SQLite replication to object storage for backup and recovery.
Articles and References
- An Introduction to WAL-on-S3 Architectures - Akihiro Okuno's 41-slide Scalar Inc. tech talk on commit placement, fencing, batching, and durability tradeoffs across databases and streaming systems. (PDF)
- Git at any scale - Design of a Git service that uses S3-compatible storage as the source of truth for repositories.
- KIP-1150: Diskless Topics - Accepted Apache Kafka proposal for topics whose durability comes from object storage.
- OSWALD - Write-ahead log design built only on object storage primitives.
- Replacing NATS with S3 - Forthcoming Rivet 3.0 design that sends latency-tolerant broadcasts through per-node S3 logs and request-reply traffic directly between nodes.
- S3 WAL Collection - Catalog of write-ahead log designs on object storage, with TLA+ specifications.
- turbopuffer Architecture - Search engine design with its write-ahead log and indexes on object storage.
- Zero-Disk S3 Storage for SQLite - Rivet's released SQLite storage, which commits to a replicated hot tier and moves idle data to S3.
Contributing
Contributions are welcome. Read the contribution guidelines first.
