Overlaybd

简体中文

Overlaybd (overlay block device) is a novel layering block-level image format, which is design for container, secure container and applicable to virtual machine. And it is an open-source implementation of paper DADI: Block-Level Image Service for Agile and Elastic Application Deployment. USENIX ATC'20".

Scaling up Without Slowing Down: Accelerating Pod Start Time. KubeCon+CloudNativeCon Europe 2024

Overlaybd is based on PhotonLibOS, which is a high-efficiency LibOS framework.

Overlaybd has 2 core component:

• Overlaybd is a block-device based image format, provideing a merged view of a sequence of block-based layers as a virtual block device. The LBA lookup algorithm employs a linearized B+ tree and AVX-512 to optimize performance, significantly accelerating search speed up to 10X. Lookup Performance

• Zfile is a compression file format which support seekalbe online decompression.

This repository is an implementation of overlaybd based on TCMU.

Overlaybd can be used as the storage backend of Accelerated Container Image, which is a solution of remote container image by fetching image data on-demand without downloading and unpacking the whole image before the container starts.

Benefits from the universality of block-device, overlaybd is also a widely applicable image format for most runtime, including qemu/kvm and any other runtime supporting block or scsi api.

Overlaybd is a non-core sub-project of containerd.

Setup

System Requirements

Overlaybd provides virtual block devices through TCMU, so the TCMU kernel module is required. TCMU is implemented in the Linux kernel and supported by most Linux distributions. For linux v6.0+, overlaybd can provides virtual block devices via UBLK.

Check and load the target_core_user module.

modprobe target_core_user ## TCMU
modprobe ublk_drv ## UBLK, linux v6.0+ requires

Install From RPM/DEB

You may download our RPM/DEB packages form Release and install.

The binaries are install to /opt/overlaybd/bin/.

Run /opt/overlaybd/bin/overlaybd-tcmu and the log is stored in /var/log/overlaybd.log.

It is better to run overlaybd-tcmu as a service so that it can be restarted after unexpected crashes.

Build From Source

Requirements

To build overlaybd from source code, the following dependencies are required:

• CMake >= 3.14

• gcc/g++ >= 8

• Libaio, libcurl, libnl3, glib2 and openssl runtime and development libraries.

  • CentOS 7/Fedora: sudo yum install libaio-devel libcurl-devel openssl-devel libnl3-devel libzstd-static e2fsprogs-devel
  • CentOS 8: sudo yum install libaio-devel libcurl-devel openssl-devel libnl3-devel libzstd-devel e2fsprogs-devel
  • Debian/Ubuntu: sudo apt install libcurl4-openssl-dev libssl-dev libaio-dev libnl-3-dev libnl-genl-3-dev libgflags-dev libzstd-dev libext2fs-dev pkg-config automake libtool # libgtest-dev // for test
  • Mariner/AzureLinux: sudo yum install libaio-devel libcurl-devel openssl-devel libnl3-devel e2fsprogs-devel glibc-devel libzstd-devel binutils ca-certificates-microsoft build-essential

Build

You need git to checkout the source code:

git clone https://github.com/containerd/overlaybd.git
cd overlaybd
git submodule update --init

The whole project is managed by CMake. Binaries and resource files will be installed to /opt/overlaybd/.

mkdir build
cd build
cmake .. # -DCMAKE_BUILD_TYPE=Debug -DCMAKE_EXPORT_COMPILE_COMMANDS=true -DBUILD_TESTING=true
make -j
sudo make install

Considering some libcurl and libopenssl has API changes, if want to build a make-sured compatible version libcurl and openssl, and link to executable as static library.

Noticed that building libcurl and openssl depends on autoconf automake and libtool.

cmake -D BUILD_CURL_FROM_SOURCE=1 ..

If you want to use the original libext2fs instead of our customized libext2fs.

cmake -D ORIGIN_EXT2FS=1 ..

For more information about ORIGIN_EXT2FS go to USERSPACE_CONVERTOR.

If you want to use DSA hardware to accelerate CRC calculation.

cmake -D ENABLE_DSA=1 ..

If you want to use avx512 to accelerate CRC calculation.

cmake -D ENABLE_ISAL=1 ..

If you want to use QAT to accelerate compression/decompression.However, currently only the decompression part has been integrated. Since LZ4 is already a highly efficient compression algorithm, our tests show that QAT can only outperform the CPU at a 4KB block size and a batch size of 256 when the compression ratio significantly exceeds a threshold.

cmake -D ENABLE_QAT=1 ..

For more information go to overlaybd/src/overlaybd/zfile/README.md.

If you want to build the ublk frontend (overlaybd-ublk), which exposes an image as /dev/ublkbN without going through TCMU/SCSI. It is built by default only when the kernel supports ublk (auto-detected from the ublk_drv module or the linux/ublk_cmd.h uapi header); force it either way with -D BUILD_UBLK_FRONTEND=on|off. Building it additionally requires autoconf, automake and libtool (liburing and libublksrv are fetched and built from source automatically).

cmake -D BUILD_UBLK_FRONTEND=on ..

Running the ublk backend requires a kernel with the ublk_drv driver (mainline

= 6.0, or a distro kernel with ublk backported). For how to run images through overlaybd-ublk / overlaybd-ublkd, see the UBLK backend section of Standalone Usage.

Finally, setup a systemd service for overlaybd-tcmu backstore.

sudo systemctl enable /opt/overlaybd/overlaybd-tcmu.service
sudo systemctl start overlaybd-tcmu

Configuration

overlaybd config

Default configure file overlaybd.json is installed to /etc/overlaybd/.

{
    "logConfig": {
        "logLevel": 1,
        "logPath": "/var/log/overlaybd.log"
    },
    "cacheConfig": {
        "cacheType": "file",
        "cacheDir": "/opt/overlaybd/registry_cache",
        "cacheSizeGB": 4
    },
    "gzipCacheConfig": {
        "enable": true,
        "cacheDir": "/opt/overlaybd/gzip_cache",
        "cacheSizeGB": 4
    },
    "credentialConfig": {
        "mode": "file",
        "path": "/opt/overlaybd/cred.json"
    },
    "ioEngine": 0,
    "workpoolSize": 8,
    "download": {
        "enable": true,
        "delay": 600,
        "delayExtra": 30,
        "maxMBps": 100
    },
    "p2pConfig": {
        "enable": false,
        "address": "localhost:19145/dadip2p"
    },
    "exporterConfig": {
        "enable": false,
        "uriPrefix": "/metrics",
        "port": 9863,
        "updateInterval": 60000000
    },
    "enableAudit": true,
    "auditPath": "/var/log/overlaybd-audit.log",
    "serviceConfig": {
        "enable": false,
        "address": "http://127.0.0.1:9862"
    }
}

NOTE: download is the config for background downloading. After an overlaybd device is lauched, a background task will be running to fetch the whole blobs into local directories. After downloading, I/O requests are directed to local files. Unlike other options, download config is reloaded when a device launching.

credential config

Important: The corresponding credential has to be set before launching devices, if the registry is not public.

Credentials are reloaded when authentication is required. Credentials have to be updated before expiration if temporary credential is used, otherwise overlaybd keeps reloading until a valid credential is set.

Overlaybd supports serveral credential mode. Here are some example credentialConfig field.

• mode file

  the credentialConfig.path should be similar to '.docker/config.json' like this:

#### /etc/overlaybd/config.json ####
{
  "logLevel": 1,
  "logPath": "/var/log/overlaybd.log",
  ...
  "credentialConfig": {
      "mode": "file",
      "path": "/opt/overlaybd/cred.json"
    },
  ...
}
#### /opt/overlaybd/cred.json ####
{
  "auths": {
    "hub.docker.com": {
      "username": "username",
      "password": "password"
    },
    "hub.docker.com/hello/world": {
      "auth": "dXNlcm5hbWU6cGFzc3dvcmQK"
    }
  }
}

• mode http

  the credentialConfig.path should be a server listening address implemented by developers and can reply to credential information.

#### /etc/overlaybd/config.json ####
{
  "logLevel": 1,
  "logPath": "/var/log/overlaybd.log",
  ...
  "credentialConfig": {
      "mode": "http",
      "path": "localhost:19876/auth"
    },
  ...
}

overlaybd will send http request to the server with remote_url like this:

GET "localhost:19876/auth?remote_url=https://hub.docker.com/v2/overlaybd/ubuntu/blobs/sha256:47e63559a8487efb55b2f1ccea9cfc04110a185c49785fdf1329d1ea462ce5f0" the server response should be formatted as follows:

{
  "traceId": "${trace_id}"
  "success": true or false
  "data": {
    "auths": {
      "hub.docker.com": {
        "username": "username",
        "password": "password"
      }
    }
  }
}

we write a sample http server in test/simple_auth_server.cpp

• mode https

  the credentialConfig.path should be an HTTPS server listening address. Unlike http mode, the https:// scheme prefix must be included in the path (e.g. https://localhost:19876/auth). The optional client_cert_path/client_key_path fields enable client certificate authentication, and server_ca_path pins trust to a specific CA. For a local auth server, providing all three fields secures communication exclusively with that server (mutual TLS).

#### /etc/overlaybd/config.json ####
{
  "logLevel": 1,
  "logPath": "/var/log/overlaybd.log",
  ...
  "credentialConfig": {
      "mode": "https",
      "path": "https://localhost:19876/auth",
      "client_cert_path": "/etc/overlaybd/client.crt",
      "client_key_path": "/etc/overlaybd/client.key",
      "server_ca_path": "/etc/overlaybd/ca.crt"
    },
  ...
}

overlaybd will send an https request with mTLS to the server with remote_url like this:

GET "https://localhost:19876/auth?remote_url=https://hub.docker.com/v2/overlaybd/ubuntu/blobs/sha256:47e63559a8487efb55b2f1ccea9cfc04110a185c49785fdf1329d1ea462ce5f0" the server response format is the same as the http mode.

All three TLS fields are optional and independently configured:

• client_cert_path sets the client certificate. If the PEM file also contains the private key, client_key_path can be omitted.

• client_key_path sets the client private key. Only needed when the key is in a separate file from the certificate.

• If server_ca_path is omitted, the system CA bundle is used to verify the server certificate. When server_ca_path is set, only the specified CA file is used — the system CA bundle is not consulted.

• mode uds

  the credentialConfig.path should be the filesystem path of a Unix-domain socket. overlaybd dials the socket and speaks the same HTTP request/response format as the http mode.

#### /etc/overlaybd/config.json ####
{
  "logLevel": 1,
  "logPath": "/var/log/overlaybd.log",
  ...
  "credentialConfig": {
      "mode": "uds",
      "path": "/run/overlaybd/creds.sock"
    },
  ...
}

overlaybd will dial the socket and send a request like:

GET "http://localhost/auth?remote_url=https://hub.docker.com/v2/overlaybd/ubuntu/blobs/sha256:47e63559a8487efb55b2f1ccea9cfc04110a185c49785fdf1329d1ea462ce5f0"

The HTTP host (localhost here) is a placeholder — the request is always dialed over the configured socket. The server response format is identical to the http mode.

Security model: a UDS is not network-reachable and is gated by filesystem permissions on the socket file. Restrict access by setting the socket owner/group and a non-world-readable mode (e.g. 0600 or 0660) on the helper side; overlaybd does not enforce this on the client.

Usage

Use with containerd

Please install overlaybd and refer to Accelerated Container Image. Overlaybd is well integrated with containerd and easy to use.

Standalone Usage

For other scenarios, users can drive overlaybd manually. An overlaybd image is exposed as a virtual block device through one of two kernel backends — TCMU (configfs, most distributions) or UBLK (/dev/ublkbN, Linux v6.0+) — and can optionally carry a writable layer.

See docs/standalone-usage.md for the config file format, the TCMU and UBLK start-up/teardown workflows, and the writable-layer (create / commit / zfile) commands.

Live Snapshot

Overlaybd can create a live snapshot without stopping the device: capture the current state of a writable layer and stack a new writable layer on top.

See docs/live-snapshot.md for the device ID, the API-service configuration and the /snapshot request/response details.

Kernel module

DADI_kmod is a kernel module of overlaybd. It can make local overlaybd-format files as a loop device or device-mapper.

Contributing

Welcome to contribute! CONTRIBUTING

Licenses

Overlaybd is released under the Apache License, Version 2.0.