datafusion-ducklake-provider

Rust crates for reading and writing DuckLake tables through Apache DataFusion.

This repository is organized as four Rust crates:

• ducklake-catalog: pure DuckLake metadata, schema, type, stats, and path logic with no async, driver, Arrow, object-store, or DataFusion dependency.
• ducklake-storage: async catalog backends and metadata maintenance hooks for SQLite, DuckDB-file, Postgres, and client-only DuckDB Quack transports.
• datafusion-ducklake-provider: DataFusion TableProvider, table factory, catalog provider, write, delete, update, and maintenance integration.
• datafusion-ducklake-cli: ducklake-cli, an interactive and scriptable SQL shell backed by DuckLakeSessionContext.

The active release surface is Rust. Python and FFI bindings are optional follow-on distribution surfaces, not current compatibility gates.

Feature Sets

ducklake-storage, datafusion-ducklake-provider, and datafusion-ducklake-cli use the same feature names:

Remote-only builds are expected to compile without DuckDB or SQLite native driver crates.

CLI

Install the SQL shell with the default SQLite backend:

cargo install datafusion-ducklake-cli
ducklake-cli --execute "SELECT 1 AS one"

Use SQL ATTACH inside the shell or scripts to connect DuckLake catalogs:

ducklake-cli --execute "ATTACH 'ducklake:sqlite:metadata.sqlite' AS lake (DATA_PATH 'data/')" \
             --execute "SELECT * FROM lake.main.numbers"

For S3-compatible data paths, install with --features object-store-s3 and pass --object-store-url plus S3 credentials through environment variables or CLI flags. See docs/cli.md.

Examples

For a full first-run walkthrough, including DataFusion-native catalog creation, an optional DuckDB-generated fixture, and the expected query output, start with docs/quickstart.md.

The provider crate ships compile-checked examples:

cargo run -p datafusion-ducklake-provider --example sqlite_table -- path/to/catalog.sqlite numbers
cargo run -p datafusion-ducklake-provider --example sqlite_catalog -- path/to/catalog.sqlite numbers
cargo run -p datafusion-ducklake-provider --example sqlite_table_factory -- path/to/catalog.sqlite numbers
cargo run -p datafusion-ducklake-provider --no-default-features --features sqlite,object-store-s3 --example s3_table_factory -- path/to/catalog.sqlite numbers
cargo run -p datafusion-ducklake-provider --no-default-features --features postgres --example postgres_catalog -- "host=localhost user=postgres dbname=ducklake" numbers
cargo run -p datafusion-ducklake-provider --no-default-features --features quack --example quack_catalog -- quack:localhost asdf numbers

The DuckDB-file example is checked separately so lightweight release gates do not repeatedly rebuild bundled DuckDB:

cargo run -p datafusion-ducklake-provider --no-default-features --features duckdb --example duckdb_file_catalog -- path/to/catalog.duckdb numbers

Compatibility Checks

See docs/compatibility.md for pinned DataFusion, Arrow, Parquet, object-store, and backend-driver versions. See docs/testing.md for the current local and CI test infrastructure.

Common local gates:

cargo fmt --check
scripts/test-fast.sh
scripts/test-coverage-only.sh

Real DuckDB, Postgres, MinIO, and Quack compatibility gates are documented in docs/testing.md and docs/integration-tests.md.

The tiered compatibility gates are:

scripts/test-upstream-direct.sh
scripts/test-local-oracle.sh
scripts/test-live-postgres-minio.sh
scripts/test-live-quack.sh
scripts/test-release.sh

scripts/test-release.sh is the full all-backend DuckLake compatibility gate, not every CI check. It is also available through the compatibility wrapper scripts/run-upstream-oracle-tests.sh. It runs sharded DuckDB-reference oracle suites for the local backends, the Postgres/MinIO live suite, the Quack live suite, and then checks that at least 90% of applicable upstream DuckLake tests have executable oracle coverage. Strict direct sqllogictest replay is still reported separately as SQL-shim coverage; it is not the DuckLake spec-compatibility percentage.

SQL Compatibility

Use datafusion_ducklake_provider::DuckLakeSessionContext when you want DuckDB-shaped DuckLake commands such as catalog ATTACH/DETACH, settings aliases, partition/sort DDL, transaction shims, and maintenance functions to work through ctx.sql(...).

use datafusion_ducklake_provider::DuckLakeSessionContext;

# async fn example() -> datafusion_ducklake_provider::datafusion::error::Result<()> {
let ctx = DuckLakeSessionContext::new();
ctx.sql(
    "ATTACH 'ducklake:sqlite:metadata.sqlite' AS lake (DATA_PATH 'data/')",
).await?;
ctx.sql("INSERT INTO lake.main.numbers VALUES (1, 'one')").await?;
ctx.sql("SELECT * FROM lake.main.numbers").await?.show().await?;
ctx.sql("DETACH lake").await?;
# Ok(())
# }

If you already own a raw DataFusion SessionContext, call ducklake_sql(&ctx, sql) for the same compatibility handling.

Supported explicit attach URI prefixes are ducklake:duckdb:, ducklake:sqlite:, ducklake:postgres:, and ducklake:quack: when the matching feature is enabled. DuckDB-file attach creates a missing catalog with the DuckDB-compatible default <metadata>.files/ data path; SQLite creation requires DATA_PATH. Postgres and Quack can also initialize missing DuckLake metadata catalogs when the remote service/user has schema/table DDL permission and DATA_PATH is supplied. Secret-backed ATTACH 'ducklake:' / ATTACH 'ducklake:<secret>' is not implemented.

For Quack, commit execution defaults to auto: use server-side DuckDB ducklake_commit when the server exposes it and the staged change shape is supported, otherwise use the client-side catalog SQL path. Set QUACK_COMMIT_MODE 'client' or QUACK_COMMIT_MODE 'server_required' in ATTACH, pass quack_commit_mode to the table factory, call QuackCatalog::with_commit_mode, or set DATAFUSION_DUCKLAKE_QUACK_COMMIT_MODE. Server-side mode stages metadata commits only; DataFusion still writes/uploads data files from the client.

The DuckDB-shaped SQL helpers dispatch through the registered DuckLake catalog backend when the matching feature is enabled. Implemented helpers include maintenance calls, metadata functions, change-feed functions, option writes, the commit-message session hook, and the documented local-file subset of ducklake_add_data_files. Remaining unsupported helpers or argument shapes are documented as clear unsupported feature errors in docs/sql-reference.md.

Documentation

• docs/README.md: documentation index and recommended reading paths.
• docs/quickstart.md: first-run DataFusion catalog creation plus query example.
• docs/user-guide.md: onboarding, dependency setup, registration patterns, backend setup, object stores, reads, writes, maintenance, and examples.
• docs/cli.md: installing and using ducklake-cli for interactive and scriptable DuckLake SQL sessions.
• docs/sql-reference.md: DuckLake-specific SQL commands, ducklake_sql compatibility shims, table function forms, DDL/DML, maintenance calls, and current unsupported SQL surface.
• docs/architecture.md: crate boundaries, read/write/maintenance paths, backend responsibilities, object-store handling, virtual columns, and testing architecture.
• docs/compatibility.md: backend, object-store, SQL, type, test-coverage, and dependency compatibility matrix.
• docs/known-limitations.md: unsupported or intentionally fail-closed current behavior.
• docs/testing.md: local test tiers, sharded oracle runners, remote service setup, logs/timings, and CI matrix structure.
• docs/integration-tests.md: local, Docker-backed, DuckDB oracle, upstream sqllogictest, and Quack test gates.
• docs/troubleshooting.md: common setup, backend, object-store, and test runner problems.

Releases And License

Release notes are tracked in CHANGELOG.md. Version tags in the form vX.Y.Z run the full CI compatibility gate and then publish ducklake-catalog, ducklake-storage, datafusion-ducklake-provider, and datafusion-ducklake-cli to crates.io in dependency order.

This project is licensed under the MIT license.