s3-orchestrator

Project Website · Documentation · Maximizing Free-Tier Storage

Most applications talk to one S3 backend, which ties them to that provider's uptime, pricing, and limits. s3-orchestrator puts a single S3 endpoint in front of any number of S3-compatible backends — OCI Object Storage, Backblaze B2, AWS S3, MinIO, Wasabi, Cloudflare R2, anything that speaks S3 — and presents them as one or more virtual buckets. It tracks where every object lives in its own database, which is what lets it enforce per-backend byte quotas, keep N copies across providers, fail reads over when one goes dark, and take a backend out of the fleet without downtime.

Clients see one endpoint and one namespace. The backends never learn the orchestrator exists — they see ordinary S3 calls, so any provider the AWS SDK can talk to works.

The web dashboard — storage summary, integrity and compression coverage, monthly usage against each provider's limits, object browser and live logs

Who this is for

What it does

• Backend quotas. Each backend carries a byte limit and writes overflow to the next when it fills, so a 20 GB allocation and a 10 GB one become one 30 GB bucket. Monthly API-request, egress and ingress caps work the same way.
• Replication. Set a factor and every object lands on that many distinct backends. A background replicator makes the copies, or write_path.parallel_copies has the write claim its targets and upload to all of them at once, answering the client on the first copy committed — which spares the replicator a full GET of the object and the source backend's egress for every copy it would have made. Reads fail over to a surviving copy, a scrubber checks stored bytes against recorded hashes, and an over-replication worker trims the set when a recovered backend brings its copies back.
• Access control. A credential resolves to a user, and that user holds a grant on each resource it may reach: a virtual bucket for object access, a backend or the instance itself for the control plane. SigV4 and presigned URLs.
• Object tagging. Key/value labels stored with the object, always on. Inline on PutObject and CreateMultipartUpload, the three ?tagging operations, and x-amz-tagging-directive on a server-side copy. Lifecycle rules can filter on a tag.
• Encryption and compression. Envelope encryption (AES-256-GCM; master key inline, in a file, or in Vault Transit) and chunked zstd compression. Both optional and transparent to clients; sizes, ETags and content hashes stay those of the object the client wrote. With both on, compression runs first, because ciphertext does not compress.
• Terraform provider. Published to the Terraform and OpenTofu registries. Manages the buckets, users, credentials and grants a deployment serves.
• Database. Embedded SQLite with no external dependencies, single-node PostgreSQL, or many instances with Redis-backed shared counters so quotas hold across the fleet.
• Operator tooling. Online drain, rebalance, import of an existing bucket, integrity scrub, a cleanup queue with a dead-letter table, hot config reload, an admin API, a web dashboard and a terminal object browser.

Moving providers without downtime

Point the orchestrator at the bucket you already have and import its objects into the metadata layer — nothing moves. Add the new provider and raise the replication factor, and the workers copy everything across while traffic keeps flowing. Once the copies are in place, drain the old backend and delete it from the config. No step takes the application down.

What else is out there

If you've gone looking for a tool that does something similar, there don't appear to be many options:

Quickstart

Prerequisites: Go 1.27+, Docker, Make.

git clone https://github.com/afreidah/s3-orchestrator.git
cd s3-orchestrator
make run

Starts three MinIO backends via Docker Compose, embedded SQLite as the metadata store, and the orchestrator on localhost:9000.

aws --endpoint-url http://localhost:9000 s3 cp /etc/hostname s3://photos/test.txt
aws --endpoint-url http://localhost:9000 s3 ls s3://photos/

Default credentials: access key photoskey, secret photossecret. Web dashboard at localhost:9000/ui/ (login admin / admin).

Full credentials and troubleshooting: docs/quickstart.md.

Install

Database: SQLite is embedded — no external dependencies for single-instance use. PostgreSQL 14+ is also an option and is required for multi-instance deployments (database.driver: postgres); the schema migrates on boot.

Generate a config interactively: s3-orchestrator init.

Verify release artifacts

Container images and release checksums are signed with cosign (keyless / Sigstore):

# Container image
cosign verify ghcr.io/afreidah/s3-orchestrator:<version> \
  --certificate-identity-regexp='github\.com/afreidah/s3-orchestrator' \
  --certificate-oidc-issuer='https://token.actions.githubusercontent.com'

# Release checksums
cosign verify-blob checksums.txt --bundle checksums.txt.bundle \
  --certificate-identity-regexp='github\.com/afreidah/s3-orchestrator' \
  --certificate-oidc-issuer='https://token.actions.githubusercontent.com'

Architecture in 30 seconds

              S3 clients (aws cli, rclone, etc.)
                          |
                          v
                    +-----------+
                    | S3 Orch.  |  <-- SigV4 auth, rate limiting, quota routing
                    +-----------+
                     |         |
            +--------+         +------------------+------------------+
            v                  v                  v                  v
       PostgreSQL        OCI Object         Backblaze B2          AWS S3
       (metadata)       Storage (20 GB)       (10 GB)             (5 GB)
                              \                  |                  /
                               '------------ 35 GB total ---------'

Metadata (object locations, quota counters, multipart state, cleanup queue) lives in PostgreSQL or SQLite. Backends only ever see plain S3 calls — no orchestrator-specific protocol, no schema requirements. Any provider that speaks the AWS SDK works.

Deeper details: docs/architecture.md.

Documentation

Contributing

Contributions welcome. Start with CONTRIBUTING.md for the build / test / submit workflow, and docs/style-guide.md for the codebase's conventions.

License

MIT