A native Rust library for Delta Lake, with bindings to Python
Python docs · Rust docs · Report a bug · Request a feature · Roadmap

Delta Lake is an open-source storage format that runs on top of existing data lakes. Delta Lake is compatible with processing engines like Apache Spark and provides benefits such as ACID transaction guarantees, schema enforcement, and scalable data handling.

The Delta Lake project aims to unlock the power of the Deltalake for as many users and projects as possible by providing native low-level APIs aimed at developers and integrators, as well as a high-level operations API that lets you query, inspect, and operate your Delta Lake with ease.

Quick Start

The deltalake library aims to adopt patterns from other libraries in data processing, so getting started should look familiar.

from deltalake import DeltaTable, write_deltalake
import pandas as pd

# write some data into a delta table
df = pd.DataFrame({"id": [1, 2], "value": ["foo", "boo"]})
write_deltalake("./data/delta", df)

# Load data from the delta table
dt = DeltaTable("./data/delta")
df2 = dt.to_pandas()

assert df.equals(df2)

The same table can also be loaded using the core Rust crate:

use deltalake::{open_table, DeltaTableError};
use url::Url;

#[tokio::main]
async fn main() -> Result<(), DeltaTableError> {
    // open the table written in python
    let delta_path = Url::from_directory_path("/abs/data/delta").unwrap();
    let table = open_table(delta_path).await?;

    // show all active files in the table
    let files: Vec<_> = table.get_file_uris()?.collect();
    println!("{files:?}");

    Ok(())
}

You can also try Delta Lake in Docker at DockerHub | Docker Repo

Get Involved

The delta-rs team is committed to providing a welcoming community.

• Join us in our Slack workspace.
• Report an issue.
• Looking to contribute?
  • Read our contributing guide.
  • See our good first issues.

Integrations

Libraries and frameworks that interoperate with delta-rs - in alphabetical order.

• AWS SDK for Pandas
• ballista
• datafusion
• Daft
• Dask
• datahub
• DuckDB
• polars
• Ray

Features

View the complete feature table.