bzfs is a CLI built for highly reliable and scalable near real-time ZFS snapshot replication with minimal operational complexity. It reliably replicates ZFS datasets in parallel using zfs send/receive and ssh, and can operate at sub-second intervals across large fleets of hosts for safe DR/HA.
Python
247
2,615 commits
updated Oct 18, 2026
bzfs is a CLI built for highly reliable and scalable near real-time ZFS snapshot replication with minimal operational complexity. It reliably replicates ZFS datasets in parallel using zfs send/receive and ssh, and can operate at sub-second intervals across large fleets of hosts. It safely supports disaster recovery and high availability (DR/HA), scale-out deployments, and protection against data loss or ransomware. It handles the many edge cases that you will eventually run into over the course of your deployment.
On the first run, bzfs replicates the source dataset and all its snapshots to the destination. On subsequent runs, it sends only changes since the previous run by incrementally replicating snapshots created on the source after that run. Source snapshots older than the most recent common snapshot on the destination are skipped automatically.
Unless bzfs is told to create snapshots or bookmarks on the source, it treats the source as read-only. With --dryrun, it
also treats the destination as read-only. In normal operation, the destination is append-only. Optional flags can delete
destination snapshots and datasets if you want to manage storage space consumption, reconcile divergence, restore production
from backup, or resync backup from production.
bzfs supports include/exclude filters that can be combined to choose which datasets, snapshots, and properties to create, replicate, delete, or compare.
A common setup uses scheduled cron jobs: one to create and prune source snapshots, one to prune destination snapshots, and
one to replicate recently created snapshots from source to destination. Schedules can run every N milliseconds, seconds,
minutes, hours, days, weeks, months, or years.
Snapshot creation, replication, pruning, monitoring, and comparison work with snapshots in any naming format, including snapshots created by third-party tools or by manual zfs snapshot commands. These functions can also be used independently.
The source can push to the destination, and the destination can pull from the source. bzfs runs on the initiator host, which
can be the source host (push mode), destination host (pull mode), same host (local mode, no network, no ssh), or a
third-party host that can SSH into source and destination (pull-push mode). In pull-push mode, the source zfs send stream
is relayed by the initiator to the destination zfs receive, without storing anything locally. For bulk data transfers,
remote-to-remote mode (--r2r=pull or --r2r=push) can instead transfer the stream directly between source and destination
to avoid making the initiator a bandwidth bottleneck. In this mode, bzfs does not need to be installed on source or
destination; only the zfs CLI is required there. bzfs can run as root or as a non-root user via sudo or delegated zfs allow permissions.
bzfs is written in Python and continuously tested with unit and integration tests on old and new ZFS versions, on multiple Linux and FreeBSD versions, and on all Python versions >= 3.9 (including latest stable, currently python-3.15).
bzfs is a stand-alone program with zero required dependencies. It is meant to run in restricted barebones server environments. No external Python packages are required; indeed no Python package management at all is required. You can symlink the program wherever you like, such as /usr/local/bin, and run it like a shell script or binary executable.
bzfs replicates snapshots for multiple datasets in parallel. It also deletes (or monitors or compares) snapshots of multiple datasets in parallel. Atomic snapshots can be created as often as every N milliseconds.
Replication progress (e.g. throughput and ETA) is shown aggregated across parallel replication tasks. Example console status line:
2025-01-17 01:23:04 [I] zfs sent 41.7 GiB 0:00:46 [963 MiB/s] [907 MiB/s] 80% ETA 0:00:04 ETA 01:23:08
bzfs uses streaming algorithms to process millions of datasets with low memory usage and low latency. It handles replication policies with multiple sources and multiple destinations per source.
Optionally, bzfs applies bandwidth rate limiting and progress monitoring (via pv) during zfs send/receive transfers. Over
the network, it can insert lightweight compression (via zstd) and buffering (via mbuffer) between endpoints. If one of
these tools is not installed, bzfs auto-detects that and continues without that auxiliary feature.
The project also ships with bzfs_jobrunner, a companion program that wraps bzfs for periodic
snapshot creation, replication, pruning, and monitoring across N source hosts and M destination hosts, using one shared
fleet-wide jobconfig script.
Typical use cases include geo-replicated backup where each destination host is in a different region and receives replicas from the same set of source hosts, low-latency replication from a primary to a secondary or to M read replicas, and backups to removable drives.
$ bzfs tank1/foo/bar dummy --recursive --skip-replication --create-src-snapshots \
--create-src-snapshots-plan "{'test':{'':{'adhoc':1}}}"
$ zfs list -t snapshot tank1/foo/bar
tank1/foo/bar@test_2024-11-06_08:30:05_adhoc
cron job:$ bzfs tank1/foo/bar dummy --recursive --skip-replication --create-src-snapshots \
--create-src-snapshots-plan \
"{'prod':{'us-west':{'minutely':40,'hourly':36,'daily':31}}}"
$ zfs list -t snapshot tank1/foo/bar
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_daily
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_hourly
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_minutely
Note: A periodic snapshot is created if it is due per the schedule indicated by its suffix (e.g. _daily or _hourly or
_minutely or _2secondly or _100millisecondly), or if the --create-src-snapshots-even-if-not-due flag is specified, or
if the most recent scheduled snapshot is somehow missing. In the latter case bzfs immediately creates a snapshot (named with
the current time, not backdated to the missed time), and then resumes the original schedule. If the suffix is _adhoc or not
a known period then a snapshot is considered non-periodic and is thus created immediately regardless of the creation time of
any existing snapshot.
$ bzfs tank1/foo/bar tank2/boo/bar
$ zfs list -t snapshot tank1/foo/bar
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_daily
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_hourly
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_minutely
$ zfs list -t snapshot tank2/boo/bar
tank2/boo/bar@prod_us-west_2024-11-06_08:30:05_daily
tank2/boo/bar@prod_us-west_2024-11-06_08:30:05_hourly
tank2/boo/bar@prod_us-west_2024-11-06_08:30:05_minutely
$ bzfs [email protected]:tank1/foo/bar tank2/boo/bar
$ bzfs tank1/foo/bar [email protected]:tank2/boo/bar
$ bzfs root@host1:tank1/foo/bar root@host2:tank2/boo/bar
--r2r=pull:$ bzfs --r2r=pull root@host1:tank1/foo/bar root@host2:tank2/boo/bar
--r2r=push:$ bzfs --r2r=push root@host1:tank1/foo/bar root@host2:tank2/boo/bar
$ bzfs --recursive tank1/foo/bar tank2/boo/bar
$ zfs list -t snapshot -r tank1/foo/bar
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_daily
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_hourly
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_minutely
tank1/foo/bar/baz@prod_us-west_2024-11-06_08:40:00_daily
tank1/foo/bar/baz@prod_us-west_2024-11-06_08:40:00_hourly
tank1/foo/bar/baz@prod_us-west_2024-11-06_08:40:00_minutely
$ zfs list -t snapshot -r tank2/boo/bar
tank2/boo/bar@prod_us-west_2024-11-06_08:30:05_daily
tank2/boo/bar@prod_us-west_2024-11-06_08:30:05_hourly
tank2/boo/bar@prod_us-west_2024-11-06_08:30:05_minutely
tank2/boo/bar/baz@prod_us-west_2024-11-06_08:40:00_daily
tank2/boo/bar/baz@prod_us-west_2024-11-06_08:40:00_hourly
tank2/boo/bar/baz@prod_us-west_2024-11-06_08:40:00_minutely
$ bzfs tank1/foo/bar tank2/boo/bar --recursive --include-snapshot-plan \
"{'prod':{'us-west':{'minutely':40,'hourly':36,'daily':31}}}"
Note: The example above compares the specified times against the standard ZFS 'creation' time property of the snapshots (which is a UTC Unix time in integer seconds), rather than against a timestamp that may be part of the snapshot name.
$ bzfs dummy tank2/boo/bar --dryrun --recursive --skip-replication \
--delete-dst-snapshots --delete-dst-snapshots-except-plan \
"{'prod':{'us-west':{'secondly':40,'minutely':40,'hourly':36,'daily':31,'weekly':12,'monthly':18,'yearly':5}}}"
Note: This also prints how many GB of disk space in total would be freed if the command were to be run for real without the --dryrun flag.
$ bzfs tank1/foo/bar tank2/boo/bar --skip-replication --compare-snapshot-lists \
--recursive --include-snapshot-regex '.*_(hourly|daily)' \
--include-snapshot-times-and-ranks '7 days ago..4 hours ago' --exclude-dataset-regex \
'(.*/)?tmp.*'
If the resulting TSV output file contains zero lines starting with the prefix 'src' and zero lines starting with the prefix 'dst' then no source snapshots are missing on the destination, and no destination snapshots are missing on the source, indicating that the periodic replication and pruning jobs perform as expected. The TSV output is sorted by dataset, and by ZFS creation time within each dataset - the first and last line prefixed with 'all' contains the metadata of the oldest and latest common snapshot, respectively. The --compare-snapshot-lists option also directly logs various summary stats, such as the metadata of the latest common snapshot, latest snapshots and oldest snapshots, as well as the time diff between the latest common snapshot and latest snapshot only in src (and only in dst), as well as how many src snapshots and how many GB of data are missing on dst, etc.
prod_us-west_<timestamp>_hourly is more than
30 minutes late (i.e. more than 30+60=90 minutes old) [warning], or more than 300 minutes late (i.e. more than 300+60=360
minutes old) [critical]. Analog for minutely and daily snapshots:$ bzfs tank1/foo/bar tank2/boo/bar --recursive --skip-replication -v --monitor-snapshots \
"{'prod':{'us-west':{'minutely':{'latest':{'warning':'30seconds','critical':'300seconds'}},'hourly':{'latest':{'warning':'30minutes','critical':'300minutes'}},'daily':{'latest':{'warning':'4hours','critical':'8hours'}}}}}"
$ bzfs tank1/foo/bar tank2/boo/bar --dryrun --recursive --force --delete-dst-datasets \
--delete-dst-snapshots
# Ubuntu / Debian:
sudo apt-get -y install zfsutils-linux python3 # ensure zfs and python are installed
sudo apt-get -y install zstd pv mbuffer # auxiliary helpers are optional
# RHEL / EL, AlmaLinux, Rocky:
sudo dnf -y install epel-release # auxiliary helpers are optional
sudo dnf -y install zstd pv mbuffer --enablerepo=epel # auxiliary helpers are optional
# FreeBSD:
pkg install -y python3 bash
pkg install -y sudo zstd pv mbuffer # auxiliary helpers are optional
# Install from git repo:
git clone https://github.com/whoschek/bzfs.git
cd bzfs/bzfs_main
./bzfs --help # Run the CLI
./bzfs_jobrunner --help
sudo ln -sf $(pwd)/bzfs /usr/local/bin/bzfs # Optional system installation
sudo ln -sf $(pwd)/bzfs_jobrunner /usr/local/bin/bzfs_jobrunner # Optional system installation
# Alternatively, install a release via pip:
python3 -m pip install bzfs
python3 -m pip install --upgrade bzfs # To upgrade an existing bzfs installation
bzfs --help # Run the CLI
bzfs_jobrunner --help
# Alternatively, setup the environment for software development:
git clone https://github.com/whoschek/bzfs.git
cd bzfs
python3 -m venv .venv # Create a Python virtual environment
. .venv/bin/activate # Activate the virtual environment
. ./preinstall_dev.sh # Prepare install
python3 -m pip install --require-hashes -r requirements-dev.txt # Install all development dependencies
python3 -m pip install -e '.[dev]' --no-deps # Make source edits take effect without reinstalling
pre-commit install --install-hooks # Set up linters/formatters to run on every commit
pre-commit run --all-files # Manually run linters/formatters
--dryrun, so they can be inspected and replayed manually.--dryrun mode, prints what would be deleted, plus estimated reclaimable disk space.sshd.zfs receive -s and zfs send -t).bzfs processes can run in parallel.snapshots_changed to reduce latency by avoiding unnecessary
zfs list -t snapshot calls.zfs send -R -s aka sending a recursive ZFS stream package.Results of continuous integration test runs on a matrix of various old and new versions of ZFS/Python/Linux/FreeBSD are here, as generated by this script. The script also demonstrates functioning installation steps on Ubuntu and FreeBSD, etc. It also generates code coverage reports which are published here.
For reproducible local testing, consider using lima_vm.sh.
On macOS or Linux, this script uses Lima to locally create a guest Ubuntu or AlmaLinux VM, then installs ZFS and runs
the bzfs test suite inside of that VM.
Also consider running the lima_testbed.sh script that uses this to create/delete a
local Lima testbed with N source VMs and M destination VMs for testing, with ZFS and VM-to-VM SSH connectivity working
out of the box. All default settings work out of the box.
Then consider running the example replication jobconfig script bzfs_job_testbed.py
on the testbed, which also works out of the box.
# Check prerequisites
zfs --version # verify ZFS is installed
python3 --version # verify Python 3.9 or newer is installed
sudo -n ls # verify non-interactive sudo works
# If sshd uses a non-standard port
# export bzfs_test_ssh_port=2222
# Choose test scope
# export bzfs_test_mode=unit # unit tests only (takes < 5 seconds)
export bzfs_test_mode=smoke # small, quick integration test subset (takes < 1 minute)
# export bzfs_test_mode=functional # most integration tests but only in a single local config (takes ~3 minutes)
# unset bzfs_test_mode # all tests (takes ~15 minutes)
export bzfs_test_no_run_quietly=true # print more than just a progress bar
# Confirm passwordless ssh works on connecting to localhost loopback address
# You should not get prompted for a password
ssh -p ${bzfs_test_ssh_port:-22} 127.0.0.1 echo hello
# Ensure tools are on remote PATH
ssh -p ${bzfs_test_ssh_port:-22} 127.0.0.1 zfs --version
ssh -p ${bzfs_test_ssh_port:-22} 127.0.0.1 zpool --version
ssh -p ${bzfs_test_ssh_port:-22} 127.0.0.1 pv --version # enables progress reporting
ssh -p ${bzfs_test_ssh_port:-22} 127.0.0.1 zstd --version # enables compression-on-the-wire
ssh -p ${bzfs_test_ssh_port:-22} 127.0.0.1 mbuffer --version # enables buffering
# Run the tests
./test.sh # when cloned from git
# or
bzfs-test # when installed via pip
Actions tab in your fork and enable GitHub Actions.All workflows -> Tests in the left sidebar.Run workflow dropdown menu on the right. The UI looks like
this screenshot.test_ubuntu_26_04 or test_ubuntu_24_04 or test_freebsd_15_1 or test_almalinux_10_zfs_2_4_hpnssh) or
choose Run all jobs to run the test suite on the full matrix of all supported platforms.usage: bzfs [-h]
[--recursive]
[--include-dataset DATASET [DATASET ...]]
[--exclude-dataset DATASET [DATASET ...]]
[--include-dataset-regex REGEX [REGEX ...]]
[--exclude-dataset-regex REGEX [REGEX ...]]
[--exclude-dataset-property STRING]
[--include-snapshot-regex REGEX [REGEX ...]]
[--exclude-snapshot-regex REGEX [REGEX ...]]
[--include-snapshot-times-and-ranks TIMERANGE [RANKRANGE ...]]
[--include-snapshot-plan DICT_STRING]
[--new-snapshot-filter-group]
[--create-src-snapshots]
[--create-src-snapshots-plan DICT_STRING]
[--create-src-snapshots-plan-attributes DICT_STRING]
[--create-src-snapshots-timeformat STRFTIME_SPEC]
[--create-src-snapshots-timezone TZ_SPEC]
[--create-src-snapshots-even-if-not-due]
[--zfs-send-program-opts STRING]
[--zfs-recv-program-opts STRING]
[--zfs-recv-program-opt STRING]
[--preserve-properties STRING [STRING ...]]
[--force-rollback-to-latest-snapshot]
[--force-rollback-to-latest-common-snapshot]
[--force]
[--force-destroy-dependents]
[--force-unmount]
[--force-once]
[--skip-parent]
[--skip-missing-snapshots [{fail,dataset,continue}]]
[--retries INT]
[--retry-min-sleep-secs FLOAT]
[--retry-initial-max-sleep-secs FLOAT]
[--retry-max-sleep-secs FLOAT]
[--retry-max-elapsed-secs FLOAT]
[--skip-on-error {fail,tree,dataset}]
[--skip-replication]
[--delete-dst-datasets]
[--delete-dst-snapshots [{snapshots,bookmarks}]]
[--delete-dst-snapshots-no-crosscheck]
[--delete-dst-snapshots-except]
[--delete-dst-snapshots-except-plan DICT_STRING]
[--delete-empty-dst-datasets [{snapshots,snapshots+bookmarks}]]
[--monitor-snapshots DICT_STRING]
[--monitor-snapshots-dont-warn]
[--monitor-snapshots-dont-crit]
[--compare-snapshot-lists [{src,dst,all,src+dst,src+all,dst+all,src+dst+all}]]
[--cache-snapshots]
[--dryrun [{recv,send}]]
[--verbose]
[--quiet]
[--no-privilege-elevation]
[--no-stream]
[--no-resume-recv]
[--create-bookmarks {all,hourly,minutely,secondly,none}]
[--no-use-bookmark]
[--ssh-cipher STRING]
[--ssh-src-user STRING]
[--ssh-dst-user STRING]
[--ssh-src-host STRING]
[--ssh-dst-host STRING]
[--ssh-src-port INT]
[--ssh-dst-port INT]
[--ssh-src-config-file FILE]
[--ssh-dst-config-file FILE]
[--ssh-control-persist-secs INT]
[--threads INT[%]]
[--max-concurrent-ssh-sessions-per-tcp-connection INT]
[--r2r {off,pull,push}]
[--bwlimit STRING]
[--no-estimate-send-size]
[--compression-program {zstd,lz4,pzstd,pigz,gzip,-}]
[--compression-program-opts STRING]
[--mbuffer-program {mbuffer,-}]
[--mbuffer-program-opts STRING]
[--ps-program {ps,-}]
[--pv-program {pv,-}]
[--pv-program-opts STRING]
[--shell-program {sh,-}]
[--ssh-program {ssh,hpnssh,-}]
[--sudo-program {sudo,doas,-}]
[--zpool-program {zpool,-}]
[--log-dir DIR]
[--log-file-prefix STRING]
[--log-file-infix STRING]
[--log-file-suffix STRING]
[--log-subdir {daily,hourly,minutely}]
[--log-syslog-address STRING]
[--log-syslog-socktype {UDP,TCP}]
[--log-syslog-facility INT]
[--log-syslog-prefix STRING]
[--log-syslog-level {CRITICAL,ERROR,WARN,INFO,DEBUG,TRACE}]
[--include-envvar-regex REGEX [REGEX ...]]
[--exclude-envvar-regex REGEX [REGEX ...]]
[--yearly_year INT]
[--yearly_month INT]
[--yearly_monthday INT]
[--yearly_hour INT]
[--yearly_minute INT]
[--yearly_second INT]
[--monthly_month INT]
[--monthly_monthday INT]
[--monthly_hour INT]
[--monthly_minute INT]
[--monthly_second INT]
[--weekly_weekday INT]
[--weekly_hour INT]
[--weekly_minute INT]
[--weekly_second INT]
[--daily_hour INT]
[--daily_minute INT]
[--daily_second INT]
[--hourly_minute INT]
[--hourly_second INT]
[--minutely_second INT]
[--secondly_millisecond INT]
[--millisecondly_microsecond INT]
[--zfs-recv-o-targets {full,incremental,full+incremental}]
[--zfs-recv-o-sources STRING]
[--zfs-recv-o-include-regex [REGEX ...]]
[--zfs-recv-o-exclude-regex REGEX [REGEX ...]]
[--zfs-recv-x-targets {full,incremental,full+incremental}]
[--zfs-recv-x-sources STRING]
[--zfs-recv-x-include-regex [REGEX ...]]
[--zfs-recv-x-exclude-regex REGEX [REGEX ...]]
[--version]
SRC_DATASET DST_DATASET [SRC_DATASET DST_DATASET ...]
SRC_DATASET DST_DATASET [SRC_DATASET DST_DATASET ...] π
SRC_DATASET: Source ZFS dataset (and its descendants) that will be replicated. Can be a ZFS filesystem or ZFS volume.
Format is [[user@]host:]dataset. The host name can also be an IPv4 address (or an IPv6 address where each ':' colon
character must be replaced with a '|' pipe character for disambiguation). If the host name is '-', the dataset will be on
the local host, and the corresponding SSH leg will be omitted. The same is true if the host is omitted and the dataset does
not contain a ':' colon at the same time. Local dataset examples: tank1/foo/bar, tank1, -:tank1/foo/bar:baz:boo
Remote dataset examples: host:tank1/foo/bar, host.example.com:tank1/foo/bar, root@host:tank,
[email protected]:tank1/foo/bar, [email protected]:tank1/foo/bar:baz:boo, user@||1:tank1/foo/bar:baz:boo. The first
component of the ZFS dataset name is the ZFS pool name, here tank1. If the option starts with a + prefix then dataset
names are read from the UTF-8 text file given after the + prefix, with each line in the file containing a SRC_DATASET and
a DST_DATASET, separated by a tab character. The basename must contain the substring 'bzfs_argument_file'. Example:
+root_dataset_names_bzfs_argument_file.txt, +/path/to/root_dataset_names_bzfs_argument_file.txt
DST_DATASET: Destination ZFS dataset for replication and deletion. Has same naming format as SRC_DATASET. During replication, destination datasets that do not yet exist are created as necessary, along with their parent and ancestors.
Performance Note: bzfs automatically replicates multiple datasets in parallel. It replicates snapshots in parallel across datasets and serially within a dataset. All child datasets of a dataset may be processed in parallel. For consistency, processing of a dataset only starts after processing of all its ancestor datasets has completed. Further, when a thread is ready to start processing another dataset, it chooses the next dataset wrt. lexicographical sort order from the datasets that are currently available for start of processing. Initially, only the roots of the selected dataset subtrees are available for start of processing. The degree of parallelism is configurable with the --threads option (see below).
-h, --help π
--recursive, -r π
--include-dataset DATASET [DATASET ...] π
During snapshot creation, replication, deletion and comparison, select any ZFS dataset (and its descendants) that is contained within SRC_DATASET (DST_DATASET in case of deletion) if its dataset name is one of the given include dataset names but none of the exclude dataset names. If a dataset is excluded its descendants are automatically excluded too, and this decision is never reconsidered even for the descendants because exclude takes precedence over include.
A dataset name is absolute if the specified dataset is prefixed by /, e.g. /tank/baz/tmp. Otherwise the dataset name is
relative wrt. source and destination, e.g. baz/tmp if the source is tank.
This option is automatically translated to an --include-dataset-regex (see below) and can be specified multiple times.
If the option starts with a + prefix then dataset names are read from the newline-separated UTF-8 text file given after
the + prefix, one dataset per line inside of the text file. The basename must contain the substring 'bzfs_argument_file'.
Examples: /tank/baz/tmp (absolute), baz/tmp (relative), +dataset_names_bzfs_argument_file.txt,
+/path/to/dataset_names_bzfs_argument_file.txt
--exclude-dataset DATASET [DATASET ...] π
--include-dataset-regex REGEX [REGEX ...] π
During snapshot creation, replication (and deletion) and comparison, select any ZFS dataset (and its descendants) that is
contained within SRC_DATASET (DST_DATASET in case of deletion) if its relative dataset path (e.g. baz/tmp) wrt.
SRC_DATASET (DST_DATASET in case of deletion) matches at least one of the given include regular expressions but none of the
exclude regular expressions. If a dataset is excluded its descendants are automatically excluded too, and this decision is
never reconsidered even for the descendants because exclude takes precedence over include.
This option can be specified multiple times. A leading ! character indicates logical negation, i.e. the regex matches if
the regex with the leading ! character removed does not match.
If the option starts with a + prefix then regex names are read from the newline-separated UTF-8 text file given after the
+ prefix, one regex per line inside of the text file. The basename must contain the substring 'bzfs_argument_file'.
Default: .* (include all datasets).
Examples: baz/tmp, (.*/)?doc[^/]*/(private|confidential).*, !public, +dataset_regexes_bzfs_argument_file.txt,
+/path/to/dataset_regexes_bzfs_argument_file.txt
--exclude-dataset-regex REGEX [REGEX ...] π
(?:.*/)?[Tt][Ee]?[Mm][Pp][-_]?[0-9]*
(exclude tmp datasets). Example: !.* (exclude no dataset)--exclude-dataset-property STRING π
The name of a ZFS dataset user property (optional). If this option is specified, the effective value (potentially inherited) of that user property is read via 'zfs list' for each selected source dataset to determine whether the dataset will be included or excluded, as follows:
a) Value is 'true' or '-' or empty string or the property is missing: Include the dataset.
b) Value is 'false': Exclude the dataset and its descendants.
c) Value is a comma-separated list of host names (no spaces, for example: 'store001,store002'): Include the dataset if the host name of the host executing bzfs is contained in the list, otherwise exclude the dataset and its descendants.
If a dataset is excluded its descendants are automatically excluded too, and the property values of the descendants are ignored because exclude takes precedence over include.
Examples: 'syncoid:sync', 'com.example.eng.project.x:backup'
Note: The use of --exclude-dataset-property is discouraged for most use cases. It is more flexible, more powerful, and more efficient to instead use a combination of --include/exclude-dataset-regex and/or --include/exclude-dataset to achieve the same or better outcome.
--include-snapshot-regex REGEX [REGEX ...] π
During replication, deletion and comparison, select any source ZFS snapshot that has a name (i.e. the part after the '@') that matches at least one of the given include regular expressions but none of the exclude regular expressions. If a snapshot is excluded this decision is never reconsidered because exclude takes precedence over include.
This option can be specified multiple times. A leading ! character indicates logical negation, i.e. the regex matches if
the regex with the leading ! character removed does not match.
Default: .* (include all snapshots). Examples: test_.*, !prod_.*, .*_(hourly|frequent), !.*_(weekly|daily)
Note: All --include/exclude-snapshot-* CLI option groups are combined into a mini filter pipeline. A filter pipeline is executed in the order given on the command line, left to right. For example if --include-snapshot-times-and-ranks (see below) is specified on the command line before --include/exclude-snapshot-regex, then --include-snapshot-times-and-ranks will be applied before --include/exclude-snapshot-regex. The pipeline results would not always be the same if the order were reversed. Order matters.
Note: During replication, bookmarks are always retained aka selected in order to help find common snapshots between source and destination.
--exclude-snapshot-regex REGEX [REGEX ...] π
--include-snapshot-times-and-ranks TIMERANGE [RANKRANGE ...] π
This option takes as input parameters a time range filter and an optional rank range filter. It separately computes the results for each filter and selects the UNION of both results. To instead use a pure rank range filter (no UNION), or a pure time range filter (no UNION), simply use 'notime' aka '0..0' to indicate an empty time range, or omit the rank range, respectively. This option can be specified multiple times.
Replication Example (UNION):
Specify to replicate all daily snapshots created during the last 7 days, and at the same time ensure that the latest 7
daily snapshots (per dataset) are replicated regardless of creation time, like so: --include-snapshot-regex '.*_daily' --include-snapshot-times-and-ranks '7 days ago..anytime' 'latest 7'
Deletion Example (no UNION):
Specify to delete all daily snapshots older than 7 days, but ensure that the latest 7 daily snapshots (per dataset) are
retained regardless of creation time, like so: --include-snapshot-regex '.*_daily' --include-snapshot-times-and-ranks notime 'all except latest 7' --include-snapshot-times-and-ranks 'anytime..7 days ago'
This helps to safely cope with irregular scenarios where no snapshots were created or received within the last 7 days, or where more than 7 daily snapshots were created within the last 7 days. It can also help to avoid accidental pruning of the last snapshot that source and destination have in common.
TIMERANGE:
The ZFS 'creation' time of a snapshot (and bookmark) must fall into this time range in order for the snapshot to be
included. The time range consists of a 'start' time, followed by a '..' separator, followed by an 'end' time. For example
'2024-01-01..2024-04-01', or 'anytime..anytime' aka *..* aka all times, or 'notime' aka '0..0' aka empty time range. Only
snapshots (and bookmarks) in the half-open time range [start, end) are included; other snapshots (and bookmarks) are
excluded. If a snapshot is excluded this decision is never reconsidered because exclude takes precedence over include. Each
of the two specified times can take any of the following forms:
a) anytime aka * wildcard; represents negative or positive infinity.
b) a non-negative integer representing a UTC Unix time in seconds. Example: 1728109805
c) an ISO 8601 datetime string with or without timezone. Examples: '2024-10-05', '2024-10-05T14:48:55', '2024-10-05T14:48:55+02', '2024-10-05T14:48:55-04:30'. If the datetime string does not contain time zone info then it is assumed to be in the local time zone. Timezone string support requires Python β₯ 3.11.
d) a duration that indicates how long ago from the current time, using the following syntax: a non-negative integer, followed by an optional space, followed by a duration unit that is one of 'seconds', 'secs', 'minutes', 'mins', 'hours', 'days', 'weeks', 'months', 'years', followed by an optional space, followed by the word 'ago'. Examples: '0secs ago', '40 mins ago', '36hours ago', '90days ago', '12weeksago'.
Note: This option compares the specified time against the standard ZFS 'creation' time property of the snapshot (which is
a UTC Unix time in integer seconds), rather than against a timestamp that may be part of the snapshot name. You can list
the ZFS creation time of snapshots and bookmarks as follows: zfs list -t snapshot,bookmark -o name,creation -s creation -d 1 $SRC_DATASET (optionally add the -p flag to display UTC Unix time in integer seconds).
Note: During replication, bookmarks are always retained aka selected in order to help find common snapshots between source and destination.
RANKRANGE:
Specifies to include the N (or N%) oldest snapshots or latest snapshots, and exclude all other snapshots (default: include no snapshots). Snapshots are sorted by creation time (actually, by the 'createtxg' ZFS property, which serves the same purpose but is more precise). The rank position of a snapshot is the zero-based integer position of the snapshot within that sorted list. A rank consists of the optional words 'all except' (followed by an optional space), followed by the word 'oldest' or 'latest', followed by a non-negative integer, followed by an optional '%' percent sign. A rank range consists of a lower rank, followed by a '..' separator, followed by a higher rank. If the optional lower rank is missing it is assumed to be 0. Examples:
'oldest 10%' aka 'oldest 0..oldest 10%' (include the oldest 10% of all snapshots)
'latest 10%' aka 'latest 0..latest 10%' (include the latest 10% of all snapshots)
'all except latest 10%' aka 'oldest 90%' aka 'oldest 0..oldest 90%' (include all snapshots except the latest 10% of all snapshots)
'oldest 90' aka 'oldest 0..oldest 90' (include the oldest 90 snapshots)
'latest 90' aka 'latest 0..latest 90' (include the latest 90 snapshots)
'all except oldest 90' aka 'oldest 90..oldest 100%' (include all snapshots except the oldest 90 snapshots)
'all except latest 90' aka 'latest 90..latest 100%' (include all snapshots except the latest 90 snapshots)
'latest 1' aka 'latest 0..latest 1' (include the latest snapshot)
'all except latest 1' aka 'latest 1..latest 100%' (include all snapshots except the latest snapshot)
'oldest 2' aka 'oldest 0..oldest 2' (include the oldest 2 snapshots)
'all except oldest 2' aka 'oldest 2..oldest 100%' (include all snapshots except the oldest 2 snapshots)
'oldest 100%' aka 'oldest 0..oldest 100%' (include all snapshots)
'oldest 0%' aka 'oldest 0..oldest 0%' (include no snapshots)
'oldest 0' aka 'oldest 0..oldest 0' (include no snapshots)
Note for multiple RANKRANGEs: --include-snapshot-times-and-ranks TIMERANGE RANKRANGE1 RANKRANGE2 is equivalent to
--include-snapshot-times-and-ranks TIMERANGE RANKRANGE1 --include-snapshot-times-and-ranks TIMERANGE RANKRANGE2.
Note: Percentage calculations are not based on the number of snapshots contained in the dataset on disk, but rather based on the number of snapshots arriving at the filter. For example, if only two daily snapshots arrive at the filter because a prior filter excludes hourly snapshots, then 'latest 10' will only include these two daily snapshots, and 'latest 50%' will only include one of these two daily snapshots.
Note: During replication, bookmarks are always retained aka selected in order to help find common snapshots between source and destination. Bookmarks do not count towards N or N% wrt. rank.
Note: If a snapshot is excluded this decision is never reconsidered because exclude takes precedence over include.
--include-snapshot-plan DICT_STRING π
Replication periods to be used if replicating snapshots within the selected destination datasets. Has the same format as --create-src-snapshots-plan and --delete-dst-snapshots-except-plan (see below). Snapshots that do not match a period will not be replicated. To avoid unexpected surprises, make sure to carefully specify ALL snapshot names and periods that shall be replicated, in combination with --dryrun.
Example: "{'prod': {'onsite': {'secondly': 40, 'minutely': 40, 'hourly': 36, 'daily': 31, 'weekly': 12, 'monthly': 18, 'yearly': 5}, 'us-west': {'secondly': 0, 'minutely': 0, 'hourly': 36, 'daily': 31, 'weekly': 12, 'monthly': 18, 'yearly': 5}, 'eu-west': {'secondly': 0, 'minutely': 0, 'hourly': 36, 'daily': 31, 'weekly': 12, 'monthly': 18, 'yearly': 5}}, 'test': {'offsite': {'12hourly': 42, 'weekly': 12}, 'onsite': {'100millisecondly': 42}}}". This example will, for the
organization 'prod' and the intended logical target 'onsite', replicate secondly snapshots that were created less than 40
seconds ago, yet replicate the latest 40 secondly snapshots regardless of creation time. Analog for the latest 40 minutely
snapshots, latest 36 hourly snapshots, etc. Note: A zero within a period (e.g. 'hourly': 0) indicates that no snapshots
shall be replicated for the given period.
Note: --include-snapshot-plan is a convenience option that auto-generates a series of the following other options: --new-snapshot-filter-group, --include-snapshot-regex, --include-snapshot-times-and-ranks
--new-snapshot-filter-group π
Starts a new snapshot filter group containing separate --{include|exclude}-snapshot-* filter options. The program separately computes the results for each filter group and selects the UNION of all results. This option can be specified multiple times and serves as a separator between groups. Example:
Delete all minutely snapshots older than 40 minutes, but ensure that the latest 40 minutely snapshots (per dataset) are
retained regardless of creation time. Additionally, delete all hourly snapshots older than 36 hours, but ensure that the
latest 36 hourly snapshots (per dataset) are retained regardless of creation time. Additionally, delete all daily snapshots
older than 31 days, but ensure that the latest 31 daily snapshots (per dataset) are retained regardless of creation time:
bzfs dummy tank2/boo/bar --dryrun --recursive --skip-replication --delete-dst-snapshots --include-snapshot-regex '.*_minutely' --include-snapshot-times-and-ranks notime 'all except latest 40' --include-snapshot-times-and-ranks 'anytime..40 minutes ago' --new-snapshot-filter-group --include-snapshot-regex '.*_hourly' --include-snapshot-times-and-ranks notime 'all except latest 36' --include-snapshot-times-and-ranks 'anytime..36 hours ago' --new-snapshot-filter-group --include-snapshot-regex '.*_daily' --include-snapshot-times-and-ranks notime 'all except latest 31' --include-snapshot-times-and-ranks 'anytime..31 days ago'
--create-src-snapshots π
Do nothing if the --create-src-snapshots flag is missing. Otherwise, before the replication step (see below), atomically create new snapshots of the source datasets selected via --{include|exclude}-dataset* policy. The names of the snapshots can be configured via --create-src-snapshots-* suboptions (see below). To create snapshots only, without any other processing such as replication, etc, consider using this flag together with the --skip-replication flag.
A periodic snapshot is created if it is due per the schedule indicated by --create-src-snapshots-plan (for example '_daily' or '_hourly' or _'10minutely' or '_2secondly' or '_100millisecondly'), or if the --create-src-snapshots-even-if-not-due flag is specified, or if the most recent scheduled snapshot is somehow missing. In the latter case bzfs immediately creates a snapshot (tagged with the current time, not backdated to the missed time), and then resumes the original schedule.
If the snapshot suffix is '_adhoc' or not a known period then a snapshot is considered non-periodic and is thus created immediately regardless of the creation time of any existing snapshot.
The implementation attempts to fit as many datasets as possible into a single (atomic) 'zfs snapshot' command line, using
lexicographical sort order, and using 'zfs snapshot -r' to the extent that this is compatible with the actual results of
the schedule and the actual results of the --{include|exclude}-dataset* pruning policy. The snapshots of all datasets that
fit within the same single 'zfs snapshot' CLI invocation will be taken within the same ZFS transaction group, and
correspondingly have identical 'createtxg' ZFS property (but not necessarily identical 'creation' ZFS time property as ZFS
actually provides no such guarantee), and thus be consistent. Dataset names that can't fit into a single command line are
spread over multiple command line invocations, respecting the limits that the operating system places on the maximum length
of a single command line, per getconf ARG_MAX.
Note: All bzfs functions including snapshot creation, replication, deletion, monitoring, comparison, etc. happily work with any snapshots in any format, even created or managed by third party ZFS snapshot management tools, including manual zfs snapshot/destroy.
--create-src-snapshots-plan DICT_STRING π
Creation periods that specify a schedule for when new snapshots shall be created on src within the selected datasets. Has the same format as --delete-dst-snapshots-except-plan.
Example: "{'prod': {'onsite': {'secondly': 40, 'minutely': 40, 'hourly': 36, 'daily': 31, 'weekly': 12, 'monthly': 18, 'yearly': 5}, 'us-west': {'secondly': 0, 'minutely': 0, 'hourly': 36, 'daily': 31, 'weekly': 12, 'monthly': 18, 'yearly': 5}, 'eu-west': {'secondly': 0, 'minutely': 0, 'hourly': 36, 'daily': 31, 'weekly': 12, 'monthly': 18, 'yearly': 5}}, 'test': {'offsite': {'12hourly': 42, 'weekly': 12}, 'onsite': {'100millisecondly': 42}}}". This example will, for the
organization 'prod' and the intended logical target 'onsite', create 'secondly' snapshots every second, 'minutely'
snapshots every minute, hourly snapshots every hour, and so on. It will also create snapshots for the targets 'us-west' and
'eu-west' within the 'prod' organization. In addition, it will create snapshots every 12 hours and every week for the
'test' organization, and name them as being intended for the 'offsite' replication target. Analog for snapshots that are
taken every 100 milliseconds within the 'test' organization.
The example creates ZFS snapshots with names like prod_onsite_<timestamp>_secondly, prod_onsite_<timestamp>_minutely,
prod_us-west_<timestamp>_hourly, prod_us-west_<timestamp>_daily, prod_eu-west_<timestamp>_hourly,
prod_eu-west_<timestamp>_daily, test_offsite_<timestamp>_12hourly, test_offsite_<timestamp>_weekly, and so on.
Note: A period name that is missing indicates that no snapshots shall be created for the given period.
The period name can contain an optional positive integer immediately preceding the time period unit, for example
_2secondly or _10minutely or _100millisecondly to indicate that snapshots are taken every 2 seconds, or every 10
minutes, or every 100 milliseconds, respectively.
--create-src-snapshots-plan-attributes DICT_STRING π
Default is "{}". Optional additional parameters that specify how to create snapshots on a per-period basis. Has the same
format as --create-src-snapshots-plan except nested values are dicts instead of integers.
Example: "{'prod': {'onsite': {'secondly': {'written_threshold': 1}}}}". This example will, for the organization 'prod'
and the intended logical target 'onsite', create a 'secondly' snapshot every second if ZFS has written at least 1 byte to
the dataset since its latest snapshot per the standard 'written' ZFS dataset property. The default for each period is
"{'written_threshold': 0}", i.e. take a snapshot even if nothing has been written. See
https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html#written
--create-src-snapshots-timeformat STRFTIME_SPEC π
Default is %Y-%m-%d_%H:%M:%S. For the strftime format, see
https://docs.python.org/3.11/library/datetime.html#strftime-strptime-behavior. Examples: %Y-%m-%d_%H:%M:%S.%f (adds
microsecond resolution), %Y-%m-%d_%H:%M:%S%z (adds timezone offset), %Y-%m-%dT%H-%M-%S (no colons).
The name of the snapshot created on the src is $org_$target_strftime(--create-src-snapshots-time*)_$period. Example:
tank/foo@prod_us-west_2024-09-03_12:26:15_daily
--create-src-snapshots-timezone TZ_SPEC π
Default is the local timezone of the system running bzfs. When creating a new snapshot on the source, fetch the current time in the specified timezone, and feed that time, and the value of --create-src-snapshots-timeformat, into the standard strftime() function to generate the timestamp portion of the snapshot name. The TZ_SPEC input parameter is of the form 'UTC' or '+HHMM' or '-HHMM' for fixed UTC offsets, or an IANA TZ identifier for auto-adjustment to daylight savings time, or the empty string to use the local timezone, for example '', 'UTC', '+0000', '+0530', '-0400', 'America/Los_Angeles', 'Europe/Vienna'. For a list of valid IANA TZ identifiers see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
To change the timezone not only for snapshot name creation, but in all respects for the entire program, use the standard
'TZ' Unix environment variable, like so: export TZ=UTC.
--create-src-snapshots-even-if-not-due π
--zfs-send-program-opts STRING π
Parameters to fine-tune 'zfs send' behaviour (optional); will be passed into 'zfs send' CLI. The value is split on runs of
one or more whitespace characters. Default is '--raw --compressed'. To run zfs send without options, specify the empty
string: --zfs-send-program-opts=''. See https://openzfs.github.io/openzfs-docs/man/master/8/zfs-send.8.html and
https://github.com/openzfs/zfs/issues/13024
If these options request a different effective raw or non-raw mode from the one recorded in an existing ZFS receive resume token, bzfs aborts the token (clears the incomplete receive state) and then proceeds as usual. ZFS may be unable to change the mode for an existing destination dataset. In that case, restore the previous send options.
--zfs-recv-program-opts STRING π
zfs receive without options, specify the empty string:
--zfs-recv-program-opts=''. Example: '-u -o canmount=noauto -o readonly=on -x keylocation -x keyformat -x encryption'.
See https://openzfs.github.io/openzfs-docs/man/master/8/zfs-receive.8.html and
https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html--zfs-recv-program-opt STRING π
--zfs-recv-program-opt=-o --zfs-recv-program-opt='org.zfsbootmenu:commandline=ro debug zswap.enabled=1'--preserve-properties STRING [STRING ...] π
On replication, preserve the current value of ZFS properties with the given names on the destination datasets. The destination ignores the property value it zfs receive's from the source if the property name matches one of the given blacklist values. This prevents a compromised or untrusted source from overwriting security-critical properties on the destination. The default is to preserve none, i.e. an empty blacklist.
Example blacklist that protects against dangerous overwrites: mountpoint overlay sharenfs sharesmb exec setuid devices encryption keyformat keylocation
See https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html and https://openzfs.github.io/openzfs-docs/man/master/8/zfs-receive.8.html#x
Note: --preserve-properties uses the 'zfs recv -x' option and thus requires either OpenZFS β₯ 2.2.0 (see https://github.com/openzfs/zfs/commit/b0269cd8ced242e66afc4fa856d62be29bb5a4ff), or that 'zfs send --props' is not used.
--force-rollback-to-latest-snapshot π
--force-rollback-to-latest-common-snapshot π
--force π
Same as --force-rollback-to-latest-common-snapshot (see above), except that additionally, if no common snapshot exists, then delete all destination snapshots before starting replication, and proceed without aborting. Without the --force* flags, the destination dataset is treated as append-only, hence no destination snapshot that already exists is deleted, and instead the operation is aborted with an error when encountering a conflicting snapshot.
Analogy: --force-rollback-to-latest-snapshot is a tiny hammer, whereas --force-rollback-to-latest-common-snapshot is a medium sized hammer, --force is a large hammer, and --force-destroy-dependents is a very large hammer. Consider using the smallest hammer that can fix the problem. No hammer is ever used by default.
--force-destroy-dependents π
--force-unmount π
--force-once, --f1 π
--skip-parent π
bzfs --recursive --skip-parent src dst is akin to Unix cp -r src/* dst/ whereas bzfs --recursive --skip-parent --skip-replication --delete-dst-datasets dummy dst is akin to Unix rm -r dst/*--skip-missing-snapshots [{fail,dataset,continue}] π
During replication, handle source datasets that select no snapshots (and no relevant bookmarks) as follows:
a) 'fail': Abort with an error.
b) 'dataset' (default): Skip the source dataset with a warning. Skip descendant datasets if --recursive and destination dataset does not exist or is inconsistent. Otherwise skip to the next dataset.
c) 'continue': Continue processing the next dataset, including descendants. If an inconsistent destination dataset prevents processing selected descendants, fail with an error; --skip-on-error determines whether processing continues elsewhere. Eventually create empty destination dataset and ancestors if they do not yet exist and source dataset has at least one descendant that selects at least one snapshot.
--retries INT π
--retry-min-sleep-secs FLOAT π
--retry-initial-max-sleep-secs FLOAT π
--retry-max-sleep-secs FLOAT π
--retry-max-elapsed-secs FLOAT π
--skip-on-error {fail,tree,dataset} π
During replication and deletion, if an error is not retryable, or --retries has been exhausted, or --skip-missing-snapshots raises an error, proceed as follows:
a) 'fail': Abort the program with an error. This mode is ideal for testing, clear error reporting, and situations where consistency trumps availability.
b) 'tree': Log the error, skip the dataset tree rooted at the dataset for which the error occurred, and continue processing the next (sibling) dataset tree. Example: Assume datasets tank/user1/foo and tank/user2/bar and an error occurs while processing tank/user1. In this case processing skips tank/user1/foo and proceeds with tank/user2.
c) 'dataset' (default): Same as 'tree' except if the destination dataset already exists and is not inconsistent, skip to the next dataset instead.
Example: Assume datasets tank/user1/foo and tank/user2/bar and an error occurs while processing tank/user1. In this case processing skips tank/user1 and proceeds with tank/user1/foo if the destination already contains tank/user1 and that dataset is not inconsistent. Otherwise processing continues with tank/user2. This mode is for production use cases that require timely forward progress even in the presence of partial failures. For example, assume the job is to backup the home directories or virtual machines of thousands of users across an organization. Even if replication of some of the datasets for some users fails due too conflicts, busy datasets, etc, the replication job will continue for the remaining datasets and the remaining users.
--skip-replication π
--delete-dst-datasets π
Do nothing if the --delete-dst-datasets option is missing. Otherwise, after successful replication step, if any, delete existing destination datasets that are selected via --{include|exclude}-dataset* policy yet do not exist within SRC_DATASET (which can be an empty dataset, such as the hardcoded virtual dataset named 'dummy'!). Do not recurse without --recursive. With --recursive, never delete non-selected dataset subtrees or their ancestors.
For example, if the destination contains datasets h1,h2,h3,d1 whereas source only contains h3, and the include/exclude policy selects h1,h2,h3,d1, then delete datasets h1,h2,d1 on the destination to make it 'the same'. On the other hand, if the include/exclude policy only selects h1,h2,h3 then only delete datasets h1,h2 on the destination to make it 'the same'.
Example to delete all tmp datasets within tank2/boo/bar: bzfs dummy tank2/boo/bar --dryrun --skip-replication --recursive --delete-dst-datasets --include-dataset-regex '(.*/)?tmp.*' --exclude-dataset-regex '!.*'
--delete-dst-snapshots [{snapshots,bookmarks}] π
Do nothing if the --delete-dst-snapshots option is missing. Otherwise, after successful replication, and successful --delete-dst-datasets step, if any, delete existing destination snapshots whose GUID does not exist within the source dataset (which can be an empty dummy dataset!) if the destination snapshots are selected by the --include/exclude-snapshot-* policy, and the destination dataset is selected via --{include|exclude}-dataset* policy. Does not recurse without --recursive.
For example, if the destination dataset contains snapshots h1,h2,h3,d1 (h=hourly, d=daily) whereas the source dataset only contains snapshot h3, and the include/exclude policy selects h1,h2,h3,d1, then delete snapshots h1,h2,d1 on the destination dataset to make it 'the same'. On the other hand, if the include/exclude policy only selects snapshots h1,h2,h3 then only delete snapshots h1,h2 on the destination dataset to make it 'the same'.
Note: To delete snapshots regardless, consider using --delete-dst-snapshots in combination with a source that is an empty
dataset, such as the hardcoded virtual dataset named 'dummy', like so: bzfs dummy tank2/boo/bar --dryrun --skip-replication --delete-dst-snapshots --include-snapshot-regex '.*_daily' --recursive
Note: Use --delete-dst-snapshots=bookmarks to delete non-temporary bookmarks instead of snapshots, in which case no snapshots are selected and the --{include|exclude}-snapshot-* filter options treat bookmarks as snapshots wrt. selecting.
Note: Does not attempt to delete snapshots that carry a zfs hold; instead auto-skips them without failing.
Performance Note: --delete-dst-snapshots operates on multiple datasets in parallel (and serially within a dataset), using the same dataset order as bzfs replication. The degree of parallelism is configurable with the --threads option (see below).
--delete-dst-snapshots-no-crosscheck π
--delete-dst-snapshots-except π
This flag indicates that the --include/exclude-snapshot-* options shall have inverted semantics for the --delete-dst-snapshots option, thus deleting all snapshots except for the selected snapshots (within the specified datasets), instead of deleting all selected snapshots (within the specified datasets). In other words, this flag enables to specify which snapshots to retain instead of which snapshots to delete.
Synchronization vs. Backup: When a real (non-dummy) source dataset is specified in combination with
--delete-dst-snapshots-except, then any destination snapshot retained by the rules above is actually only retained if it
also exists in the source dataset - all other destination snapshots are deleted. This is great for synchronization use
cases but should NEVER BE USED FOR LONG-TERM ARCHIVAL. Long-term archival use cases should instead specify the dummy
source dataset as they require an independent retention policy that is not tied to the current contents of the source
dataset.
--delete-dst-snapshots-except-plan DICT_STRING π
Retention periods to be used if pruning snapshots or bookmarks within the selected destination datasets via --delete-dst-snapshots. Has the same format as --create-src-snapshots-plan. Snapshots (--delete-dst-snapshots=snapshots) or bookmarks (with --delete-dst-snapshots=bookmarks) that do not match a period will be deleted. To avoid unexpected surprises, make sure to carefully specify ALL snapshot names and periods that shall be retained, in combination with --dryrun.
Example: "{'prod': {'onsite': {'secondly': 40, 'minutely': 40, 'hourly': 36, 'daily': 31, 'weekly': 12, 'monthly': 18, 'yearly': 5}, 'us-west': {'secondly': 0, 'minutely': 0, 'hourly': 36, 'daily': 31, 'weekly': 12, 'monthly': 18, 'yearly': 5}, 'eu-west': {'secondly': 0, 'minutely': 0, 'hourly': 36, 'daily': 31, 'weekly': 12, 'monthly': 18, 'yearly': 5}}, 'test': {'offsite': {'12hourly': 42, 'weekly': 12}, 'onsite': {'100millisecondly': 42}}}". This example will, for the
organization 'prod' and the intended logical target 'onsite', retain secondly snapshots that were created less than 40
seconds ago, yet retain the latest 40 secondly snapshots regardless of creation time. Analog for the latest 40 minutely
snapshots, latest 36 hourly snapshots, etc. It will also retain snapshots for the targets 'us-west' and 'eu-west' within
the 'prod' organization. In addition, within the 'test' organization, it will retain snapshots that are created every 12
hours and every week as specified, and name them as being intended for the 'offsite' replication target. Analog for
snapshots that are taken every 100 milliseconds within the 'test' organization. All other snapshots within the selected
datasets will be deleted - you've been warned!
The example scans the selected ZFS datasets for snapshots with names like prod_onsite_<timestamp>_secondly,
prod_onsite_<timestamp>_minutely, prod_us-west_<timestamp>_hourly, prod_us-west_<timestamp>_daily,
prod_eu-west_<timestamp>_hourly, prod_eu-west_<timestamp>_daily, test_offsite_<timestamp>_12hourly,
test_offsite_<timestamp>_weekly, and so on, and deletes all snapshots that do not match a retention rule.
Note: A zero within a period (e.g. 'hourly': 0) indicates that no snapshots shall be retained for the given period.
Note: --delete-dst-snapshots-except-plan is a convenience option that auto-generates a series of the following other options: --delete-dst-snapshots-except, --new-snapshot-filter-group, --include-snapshot-regex, --include-snapshot-times-and-ranks
--delete-empty-dst-datasets [{snapshots,snapshots+bookmarks}] π
Do nothing if the --delete-empty-dst-datasets option is missing or --recursive is missing. Otherwise, after successful replication step and successful --delete-dst-datasets and successful --delete-dst-snapshots steps, if any, delete any selected destination dataset that has no snapshot and no bookmark if all descendants of that destination dataset are also selected and do not have a snapshot or bookmark either (again, only if the existing destination dataset is selected via --{include|exclude}-dataset* policy). Never delete non-selected dataset subtrees or their ancestors.
For example, if the destination contains datasets h1,d1, and the include/exclude policy selects h1,d1, then check if h1,d1 can be deleted. On the other hand, if the include/exclude policy only selects h1 then only check if h1 can be deleted.
Note: Use --delete-empty-dst-datasets=snapshots to delete snapshot-less datasets even if they still contain bookmarks.
--monitor-snapshots DICT_STRING π
Do nothing if the --monitor-snapshots flag is missing. Otherwise, after all other steps, alert the user if the ZFS
'creation' time property of the latest snapshot for any specified snapshot name pattern within the selected datasets is too
old wrt. the specified age limit. The purpose is to check if snapshots are successfully taken on schedule, successfully
replicated on schedule, and successfully pruned on schedule. Process exit code is 0, 1, 2 on OK, WARNING, CRITICAL,
respectively. Example DICT_STRING: "{'prod': {'onsite': {'100millisecondly': {'latest': {'warning': '300 milliseconds', 'critical': '2 seconds'}}, 'secondly': {'latest': {'warning': '2 seconds', 'critical': '14 seconds'}}, 'minutely': {'latest': {'warning': '30 seconds', 'critical': '300 seconds'}}, 'hourly': {'latest': {'warning': '30 minutes', 'critical': '300 minutes'}}, 'daily': {'latest': {'warning': '4 hours', 'critical': '8 hours'}}, 'weekly': {'latest': {'warning': '2 days', 'critical': '8 days'}}, 'monthly': {'latest': {'warning': '2 days', 'critical': '8 days'}}, 'yearly': {'latest': {'warning': '5 days', 'critical': '14 days'}}, '10minutely': {'latest': {'warning': '0 minutes', 'critical': '0 minutes'}}}, '': {'daily': {'latest': {'warning': '4 hours', 'critical': '8 hours'}}}}}". This example alerts the user if
the latest src or dst snapshot named prod_onsite_<timestamp>_hourly is more than 30 minutes late (i.e. more than 30+60=90
minutes old) [warning] or more than 300 minutes late (i.e. more than 300+60=360 minutes old) [critical]. Analog for the
latest snapshot named prod_<timestamp>_daily, and so on.
Note: A duration that is missing or zero (e.g. '0 minutes') indicates that no snapshots shall be checked for the given snapshot name pattern.
Example output with --verbose:
--monitor_snapshots: OK. Latest snapshot for tank/foo@prod_<timestamp>_daily is 4.18h old: @prod_2025-01-10_08:30:05_daily
--monitor_snapshots: OK. Latest snapshot for tank/bar@prod_<timestamp>_daily is 4.18h old: @prod_2025-01-10_08:30:05_daily
--monitor_snapshots: Latest snapshot for tank/baz@prod_<timestamp>_daily is 1.2d old but should be at most 1.1d old: @prod_2025-01-09_08:30:05_daily
...
ERROR: Exiting bzfs with status code 2. Cause: --monitor_snapshots: Latest snapshot for tank/baz@prod_<timestamp>_daily is 1.2d old but should be at most 1.1d old: @prod_2025-01-09_08:30:05_daily
--monitor-snapshots-dont-warn π
--monitor-snapshots-dont-crit π
--compare-snapshot-lists [{src,dst,all,src+dst,src+all,dst+all,src+dst+all}] π
Do nothing if the --compare-snapshot-lists option is missing. Otherwise, after successful replication step and successful --delete-dst-datasets, --delete-dst-snapshots steps and --delete-empty-dst-datasets steps, if any, proceed as follows:
Compare source and destination dataset trees recursively wrt. snapshots, for example to check if all recently taken snapshots have been successfully replicated by a periodic job.
Example: List snapshots only contained in source (tagged with 'src'), only contained in destination (tagged with 'dst'),
and contained in both source and destination (tagged with 'all'), restricted to hourly and daily snapshots taken within the
last 7 days, excluding the last 4 hours (to allow for some slack/stragglers), excluding temporary datasets: bzfs tank1/foo/bar tank2/boo/bar --skip-replication --compare-snapshot-lists=src+dst+all --recursive --include-snapshot-regex '.*_(hourly|daily)' --include-snapshot-times-and-ranks '7 days ago..4 hours ago' --exclude-dataset-regex 'tmp.*'
This outputs a TSV file containing the following columns:
location creation_iso createtxg rel_name guid root_dataset rel_dataset name creation written
Example output row:
src 2024-11-06_08:30:05 17435050 /foo@test_2024-11-06_08:30:05_daily 2406491805272097867 tank1/src /foo tank1/src/foo@test_2024-10-06_08:30:04_daily 1730878205 24576
If the TSV output file contains zero lines starting with the prefix 'src' and zero lines starting with the prefix 'dst' then no source snapshots are missing on the destination, and no destination snapshots are missing on the source, indicating that the periodic replication and pruning jobs perform as expected. The TSV output is sorted by rel_dataset, and by ZFS creation time within each rel_dataset - the first and last line prefixed with 'all' contains the metadata of the oldest and latest common snapshot, respectively. Third party tools can use this info for post-processing, for example using custom scripts using 'csplit' or duckdb analytics queries.
The --compare-snapshot-lists option also directly logs various summary stats, such as the metadata of the latest common snapshot, latest snapshots and oldest snapshots, as well as the time diff between the latest common snapshot and latest snapshot only in src (and only in dst), as well as how many src snapshots and how many GB of data are missing on dst, etc.
Note: By default, if the source ZFS pool supports bookmarks, source bookmarks also participate in the comparison. A source bookmark and destination snapshot with the same GUID are considered to be contained in both locations, even if the corresponding source snapshot has already been deleted. Unmatched temporary bookmarks managed by bzfs remain hidden. Specify --no-use-bookmark to compare snapshots only.
Note: Consider omitting the 'all' flag to reduce noise and instead focus on missing snapshots only, like so: --compare-snapshot-lists=src+dst
Note: The source can also be an empty dataset, such as the hardcoded virtual dataset named 'dummy'.
Note: --compare-snapshot-lists is typically much faster than standard 'zfs list -t snapshot' CLI usage because the former issues requests with a higher degree of parallelism than the latter. The degree is configurable with the --threads option (see below).
--cache-snapshots π
If --cache-snapshots is specified, maintain a persistent local cache of recent snapshot creation times, recent successful replication times, and recent monitoring times, and compare them to a quick 'zfs list -t filesystem,volume -p -o snapshots_changed' to help determine if a new snapshot shall be created on the src, and if there are any changes that need to be replicated or monitored. Enabling the cache improves performance if --create-src-snapshots and/or replication and/or --monitor-snapshots is invoked frequently (e.g. every minute via cron) over a large number of datasets, with each dataset containing a large number of snapshots, yet it is seldom for a new src snapshot to actually be created, or there are seldom any changes to replicate or monitor (e.g. a snapshot is only created every day and/or deleted every day).
Note: This flag only has an effect on OpenZFS β₯ 2.2.
Note: This flag is only relevant for snapshot creation on the src if --create-src-snapshots-even-if-not-due is not specified.
--dryrun [{recv,send}], -n [{recv,send}] π
Do a dry run (aka 'no-op') to print what operations would happen if the command were to be executed for real (optional). This option treats both the ZFS source and destination as read-only. Accepts an optional argument for fine tuning that is handled as follows:
a) 'recv': Send snapshot data via 'zfs send' to the destination host and receive it there via 'zfs receive -n', which discards the received data there.
b) 'send': Do not execute 'zfs send' and do not execute 'zfs receive'. This is a less 'realistic' form of dry run, but much faster, especially for large snapshots and slow networks/disks, as no snapshot is actually transferred between source and destination. This is the default when specifying --dryrun.
Examples: --dryrun, --dryrun=send, --dryrun=recv
--verbose, -v π
-v -v -v flag, maybe along with --dryrun. All ZFS and
SSH commands (even with --dryrun) are logged such that they can be inspected, copy-and-pasted into a terminal shell and run
manually to help anticipate or diagnose issues. ERROR, WARN, INFO, DEBUG, TRACE output lines are identified by [E], [W],
[I], [D], [T] prefixes, respectively.--quiet, -q π
--no-privilege-elevation, -p π
Do not attempt to run state changing ZFS operations 'zfs create/rollback/destroy/send/receive/snapshot' as root (via 'sudo
-u root' elevation granted by administrators appending the following to /etc/sudoers: <NON_ROOT_USER_NAME> ALL=NOPASSWD:/path/to/zfs
Instead, the --no-privilege-elevation flag is for non-root users that have been granted corresponding ZFS permissions by administrators via 'zfs allow' delegation mechanism, like so: sudo zfs allow -u $SRC_NON_ROOT_USER_NAME snapshot,destroy,send,bookmark,hold $SRC_DATASET; sudo zfs allow -u $DST_NON_ROOT_USER_NAME mount,create,receive,rollback,destroy $DST_DATASET_OR_POOL.
If you do not plan to use the --force* flags and --delete-* CLI options then ZFS permissions 'rollback,destroy' can be
omitted, arriving at the absolutely minimal set of required destination permissions: mount,create,receive.
For extra security $SRC_NON_ROOT_USER_NAME should be different than $DST_NON_ROOT_USER_NAME, i.e. the sending Unix user on the source and the receiving Unix user at the destination should be separate Unix user accounts with separate private keys even if both accounts reside on the same machine, per the principle of least privilege.
Also see https://openzfs.github.io/openzfs-docs/man/master/8/zfs-allow.8.html#EXAMPLES and https://tinyurl.com/9h97kh8n and https://youtu.be/o_jr13Z9f1k?si=7shzmIQJpzNJV6cq
--no-stream π
--no-resume-recv π
Replication of snapshots via 'zfs send/receive' can be interrupted by intermittent network hiccups, reboots, hardware issues, etc. Interrupted 'zfs send/receive' operations are retried if the --retries and --retry-* options enable it (see above). In normal operation bzfs automatically retries such that only the portion of the snapshot is transmitted that has not yet been fully received on the destination. For example, this helps to progressively transfer a large individual snapshot over a wireless network in a timely manner despite frequent intermittent network hiccups. This optimization is called 'resume receive' and uses the 'zfs receive -s' and 'zfs send -t' feature.
The --no-resume-recv option disables this optimization such that a retry now retransmits the entire snapshot from scratch, which could slow down or even prohibit progress in case of frequent network hiccups. bzfs automatically falls back to using the --no-resume-recv option if it is auto-detected that the ZFS pool does not reliably support the 'resume receive' optimization.
Note: Snapshots that have already been fully transferred as part of the current 'zfs send/receive' operation need not be retransmitted regardless of the --no-resume-recv flag. For example, assume a single 'zfs send/receive' operation is transferring incremental snapshots 1 through 10 via 'zfs send -I', but the operation fails while transferring snapshot 10, then snapshots 1 through 9 need not be retransmitted regardless of the --no-resume-recv flag, as these snapshots have already been successfully received at the destination either way.
--create-bookmarks {all,hourly,minutely,secondly,none} π
For increased safety, bzfs replication behaves as follows wrt. ZFS bookmark creation, if it is autodetected that the source ZFS pool supports bookmarks:
all (default): Selects every source snapshot that will be sent during each 'zfs send' operation. This increases safety
at the expense of a little performance.
hourly: Selects each hourly, daily, weekly, monthly and yearly source snapshot that will be sent.
minutely and secondly: Same as hourly except that it also selects minutely and secondly snapshots, respectively.
none: No bookmark is created.
For every mode except none, the snapshot of a full send and the final snapshot of incremental replication are always
selected as they establish the initial and latest common bases, respectively.
For each selected source snapshot, bzfs creates a destination-specific temporary bookmark before starting its 'zfs send/receive' operation. After the operation succeeds, it renames the temporary bookmark to a finalized bookmark and deletes any obsolete temporary bookmarks for that destination. Both temporary and finalized bookmarks can serve as sources for future incremental replication once the corresponding snapshot exists on the destination. This guarantees continuity even if the bzfs process is killed after 'zfs send/receive' succeeds but before bookmark finalization.
Bookmarks exist so an incremental stream can continue to be sent from the source dataset without having to keep the already replicated snapshot around on the source dataset until the next upcoming snapshot has been successfully replicated. This way you can bookmark a snapshot on the source dataset, send the snapshot to another host, then delete the snapshot from the source dataset to save disk space, and then still incrementally send the next upcoming snapshot from the source dataset to the other host by referring to the bookmark.
The --create-bookmarks=none option disables this safety feature but is discouraged, because bookmarks are tiny and relatively cheap and help to ensure that ZFS replication can continue even if source and destination dataset somehow have no common snapshot anymore. For example, if a pruning script has accidentally deleted too many (or even all) snapshots on the source dataset in an effort to reclaim disk space, replication can still proceed because it can use the info in the bookmark (the bookmark must still exist in the source dataset) instead of the info in the metadata of the (now missing) source snapshot.
A ZFS bookmark is a tiny bit of metadata extracted from a ZFS snapshot by the 'zfs bookmark' CLI, and attached to a dataset, much like a ZFS snapshot. Note that a ZFS bookmark does not contain user data; instead a ZFS bookmark is essentially a tiny pointer in the form of the GUID of the snapshot and 64-bit transaction group number of the snapshot and creation time of the snapshot, which is sufficient to tell the destination ZFS pool how to find the destination snapshot corresponding to the source bookmark and (potentially already deleted) source snapshot. A bookmark can be fed into 'zfs send' as the source of an incremental send. Note that while a bookmark allows for its snapshot to be deleted on the source after successful replication, it still requires that its snapshot is not somehow deleted prematurely on the destination dataset, so be mindful of that. Also see https://www.youtube.com/watch?v=LaNgoAZeTww&t=316s.
By convention, a finalized bookmark created by bzfs has the same name as its corresponding snapshot, except the name also
contains the snapshot GUID for uniqueness. Destination-specific temporary bookmarks use names that begin with .TMPBZFS.;
this namespace is reserved for internal use. Do not create or delete such temporary bookmarks because bzfs automatically
manages and garbage collects them.
You can list bookmarks, like so: zfs list -t bookmark -o name,guid,createtxg,creation -d 1 $SRC_DATASET, and you can (and
should) periodically prune obsolete bookmarks just like snapshots, like so: zfs destroy $SRC_DATASET#$BOOKMARK.
Typically, bookmarks should be pruned less aggressively than snapshots, and destination snapshots should be pruned less
aggressively than source snapshots. As an example starting point, here is a command that deletes all non-temporary
bookmarks older than 90 days, but retains the latest 200 non-temporary bookmarks (per dataset) regardless of creation time:
bzfs dummy tank2/boo/bar --dryrun --recursive --skip-replication --delete-dst-snapshots=bookmarks --include-snapshot-times-and-ranks notime 'all except latest 200' --include-snapshot-times-and-ranks 'anytime..90 days ago'
--no-use-bookmark π
For increased safety, in normal replication operation bzfs replication also looks for bookmarks (in addition to snapshots) on the source dataset in order to find the most recent common snapshot wrt. the destination dataset, if it is auto-detected that the source ZFS pool supports bookmarks. The --no-use-bookmark option disables this safety feature but is discouraged, because bookmarks help to ensure that ZFS replication can continue even if source and destination dataset somehow have no common snapshot anymore.
Note that it does not matter whether a bookmark was created by bzfs or a third party script, as only the GUID of the
bookmark and the GUID of the snapshot is considered for comparison, and ZFS guarantees that any bookmark of a given
snapshot automatically has the same GUID, transaction group number and creation time as the snapshot. Apart from the
internal .TMPBZFS. namespace described above, you can create, delete and prune bookmarks any way you like, as bzfs
(without --no-use-bookmark) will happily work with whatever bookmarks currently exist, if any.
--ssh-cipher STRING π
^[email protected]--ssh-src-user STRING π
--ssh-dst-user STRING π
--ssh-src-host STRING π
--ssh-dst-host STRING π
--ssh-src-port INT π
--ssh-dst-port INT π
--ssh-src-config-file FILE π
--ssh-dst-config-file FILE π
--ssh-control-persist-secs INT π
--threads INT[%] π
--max-concurrent-ssh-sessions-per-tcp-connection INT π
For best throughput, bzfs uses multiple SSH TCP connections in parallel, as indicated by --threads (see above). For best startup latency, each such parallel TCP connection can carry a maximum of S concurrent SSH sessions, where S=--max-concurrent-ssh-sessions-per-tcp-connection (default: 8, min: 1). Concurrent SSH sessions are mostly used for metadata operations such as listing ZFS datasets and their snapshots. This client-side max sessions parameter must not be higher than the server-side sshd_config(5) MaxSessions parameter (which defaults to 10, see https://manpages.ubuntu.com/manpages/latest/man5/sshd_config.5.html).
Note: For better throughput, bzfs uses one dedicated TCP connection per ZFS send/receive operation such that the dedicated connection is never used by any other concurrent SSH session, effectively ignoring the value of the --max-concurrent-ssh-sessions-per-tcp-connection parameter in the ZFS send/receive case.
--r2r {off,pull,push} π
For remote-to-remote replication, controls whether the zfs send stream is relayed by the initiator or transferred
directly between source and destination in order to improve the throughput of bulk data transfers (default: off). This
option has no effect unless both replication endpoints are remote.
off: Keeps existing pull-push behavior where the initiator acts as an intermediary that relays the zfs send stream
between source and destination, which can become a central bandwidth bottleneck.Example: ssh alice@srchost 'zfs send ...' | ssh bob@dsthost 'zfs receive ...'
pull: Tells the destination host to pull the zfs send stream directly from the source host. Requires sh and ssh
on the destination host, plus SSH setup such that the destination user@host can ssh into the source host.Example: ssh bob@dsthost "sh -c 'ssh alice@srchost zfs send ... | zfs receive ...'"
push: Tells the source host to push the zfs send stream directly to the destination host. Requires sh and ssh on
the source host, plus SSH setup such that the source user@host can ssh into the destination host.Example: ssh alice@srchost "sh -c 'zfs send ... | ssh bob@dsthost zfs receive ...'"
Note: Orchestration still runs on the initiator; only the bulk data path changes. It is recommended to also set
--ssh-src-user and --ssh-dst-user explicitly in order to avoid potential confusion about which SSH user account
performs the nested remote-to-remote operation.
Note: If the required nested sh/ssh do not exist, bzfs falls back to --r2r=off. --r2r=pull falls back to
--r2r=off if --ssh-src-config-file is set to a non-empty value other than none, and --r2r=push falls back to
--r2r=off if --ssh-dst-config-file is set to a non-empty value other than none.
Note: --pv* progress reporting options have no effect in r2r modes; progress reporting is disabled.
--bwlimit STRING π
pv and mbuffer bandwidth rate limit for zfs send/receive data transfer (optional). Example: 100m to cap
throughput at 100 MB/sec. Default is unlimited. Also see https://manpages.ubuntu.com/manpages/latest/man1/pv.1.html--no-estimate-send-size π
--compression-program {zstd,lz4,pzstd,pigz,gzip,-} π
--compression-program-opts STRING π
--mbuffer-program {mbuffer,-} π
--mbuffer-program-opts STRING π
--ps-program {ps,-} π
--pv-program {pv,-} π
--pv-program-opts STRING π
--shell-program {sh,-} π
--ssh-program {ssh,hpnssh,-} π
--sudo-program {sudo,doas,-} π
--zpool-program {zpool,-} π
--log-dir DIR π
tail --follow=name --max-unchanged-stats=1 on both symlinks to follow what's
currently going on. Parallel replication generates a separate .pv file per thread. To monitor these, run something like
while true; do clear; for f in $(realpath $HOME/bzfs-logs/current/current.pv)*; do tac -s $(printf '\r') $f | tr '\r' '\n' | grep -m1 -v '^$'; done; sleep 1; done--log-file-prefix STRING π
${--log-dir}/${--log-file-prefix}<timestamp>${--log-file-infix}${--log-file-suffix}-<random>.log. Example:
--log-file-prefix=zrun_us-west_ --log-file-suffix=_daily will generate log file names such as
zrun_us-west_2024-09-03_12:26:15_daily-bl4i1fth.log--log-file-infix STRING π
${--log-dir}/${--log-file-prefix}<timestamp>${--log-file-infix}${--log-file-suffix}-<random>.log. Example:
--log-file-prefix=zrun_us-west_ --log-file-suffix=_daily will generate log file names such as
zrun_us-west_2024-09-03_12:26:15_daily-bl4i1fth.log--log-file-suffix STRING π
${--log-dir}/${--log-file-prefix}<timestamp>${--log-file-infix}${--log-file-suffix}-<random>.log. Example:
--log-file-prefix=zrun_us-west_ --log-file-suffix=_daily will generate log file names such as
zrun_us-west_2024-09-03_12:26:15_daily-bl4i1fth.log--log-subdir {daily,hourly,minutely} π
--log-syslog-address STRING π
--log-syslog-socktype {UDP,TCP} π
--log-syslog-facility INT π
--log-syslog-prefix STRING π
--log-syslog-level {CRITICAL,ERROR,WARN,INFO,DEBUG,TRACE} π
--include-envvar-regex REGEX [REGEX ...] π
On program startup, unset all Unix environment variables for which the full environment variable name matches at least one of the excludes but none of the includes. If an environment variable is included this decision is never reconsidered because include takes precedence over exclude. The purpose is to tighten security and help guard against accidental inheritance or malicious injection of environment variable values that may have unintended effects.
This option can be specified multiple times. A leading ! character indicates logical negation, i.e. the regex matches if
the regex with the leading ! character removed does not match. The default is to include no environment variables, i.e.
to make no exceptions to --exclude-envvar-regex. Example that retains at least these two env vars: --include-envvar-regex PATH --include-envvar-regex bzfs_min_pipe_transfer_size. Example that retains all environment variables without tightened
security: '.*'
--exclude-envvar-regex REGEX [REGEX ...] π
bzfs_.*--version π
Use these options to customize when snapshots that happen every N years are scheduled to be created on the source by the --create-src-snapshots option.
--yearly_year INT π
--yearly_month INT π
--yearly_monthday INT π
--yearly_hour INT π
--yearly_minute INT π
--yearly_second INT π
Use these options to customize when snapshots that happen every N months are scheduled to be created on the source by the --create-src-snapshots option.
--monthly_month INT π
--monthly_monthday INT π
--monthly_hour INT π
--monthly_minute INT π
--monthly_second INT π
Use these options to customize when snapshots that happen every N weeks are scheduled to be created on the source by the --create-src-snapshots option.
--weekly_weekday INT π
--weekly_hour INT π
--weekly_minute INT π
--weekly_second INT π
Use these options to customize when snapshots that happen every N days are scheduled to be created on the source by the --create-src-snapshots option.
--daily_hour INT π
--daily_minute INT π
--daily_second INT π
Use these options to customize when snapshots that happen every N hours are scheduled to be created on the source by the --create-src-snapshots option.
--hourly_minute INT π
--hourly_second INT π
Use these options to customize when snapshots that happen every N minutes are scheduled to be created on the source by the --create-src-snapshots option.
--minutely_second INT π
Use these options to customize when snapshots that happen every N seconds are scheduled to be created on the source by the --create-src-snapshots option.
--secondly_millisecond INT π
Use these options to customize when snapshots that happen every N milliseconds are scheduled to be created on the source by the --create-src-snapshots option.
--millisecondly_microsecond INT π
The following group of parameters specifies additional zfs receive '-o' options that can be used to configure copying of ZFS dataset properties from the source dataset to its corresponding destination dataset. The 'zfs-recv-o' group of parameters is applied before the 'zfs-recv-x' group.
--zfs-recv-o-targets {full,incremental,full+incremental} π
--zfs-recv-o-sources STRING π
--zfs-recv-o-include-regex [REGEX ...] π
Take the output properties of --zfs-recv-o-sources (see above) and filter them such that we only retain the properties whose name matches at least one of the --include regexes but none of the --exclude regexes. If a property is excluded this decision is never reconsidered because exclude takes precedence over include. Append each retained property to the list of '-o' options in --zfs-recv-program-opt(s), unless another '-o' or '-x' option with the same name already exists therein. In other words, --zfs-recv-program-opt(s) takes precedence.
Zero or more regexes can be specified. Specify zero regexes to append no extra '-o' option. A leading ! character
indicates logical negation, i.e. the regex matches if the regex with the leading ! character removed does not match. If
the option starts with a + prefix then regexes are read from the newline-separated UTF-8 text file given after the +
prefix, one regex per line inside of the text file. The basename must contain the substring 'bzfs_argument_file'.
The default regex is
'aclinherit|aclmode|acltype|atime|checksum|compression|copies|logbias|primarycache|recordsize|redundant_metadata|relatime|secondarycache|snapdir|sync|xattr'.
Example: --zfs-recv-o-include-regex compression recordsize. More examples: .* (include all properties), foo bar myapp:.* (include three regexes) +zfs-recv-o_regexes_bzfs_argument_file.txt,
+/path/to/zfs-recv-o_regexes_bzfs_argument_file.txt
See https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html
--zfs-recv-o-exclude-regex REGEX [REGEX ...] π
The following group of parameters specifies additional zfs receive '-x' options that can be used to configure copying of ZFS dataset properties from the source dataset to its corresponding destination dataset. The 'zfs-recv-o' group of parameters is applied before the 'zfs-recv-x' group.
--zfs-recv-x-targets {full,incremental,full+incremental} π
--zfs-recv-x-sources STRING π
--zfs-recv-x-include-regex [REGEX ...] π
Take the output properties of --zfs-recv-x-sources (see above) and filter them such that we only retain the properties whose name matches at least one of the --include regexes but none of the --exclude regexes. If a property is excluded this decision is never reconsidered because exclude takes precedence over include. Append each retained property to the list of '-x' options in --zfs-recv-program-opt(s), unless another '-o' or '-x' option with the same name already exists therein. In other words, --zfs-recv-program-opt(s) takes precedence.
Zero or more regexes can be specified. Specify zero regexes to append no extra '-x' option. A leading ! character
indicates logical negation, i.e. the regex matches if the regex with the leading ! character removed does not match. If
the option starts with a + prefix then regexes are read from the newline-separated UTF-8 text file given after the +
prefix, one regex per line inside of the text file. The basename must contain the substring 'bzfs_argument_file'.
The default is to include no properties, thus by default no extra '-x' option is appended. Example:
--zfs-recv-x-include-regex compression recordsize. More examples: .* (include all properties), foo bar myapp:.*
(include three regexes) +zfs-recv-x_regexes_bzfs_argument_file.txt, +/path/to/zfs-recv-x_regexes_bzfs_argument_file.txt
See https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html
--zfs-recv-x-exclude-regex REGEX [REGEX ...] π
2,247 followers Β· starred Apr 2026
185 followers Β· starred Jan 2026
2,273 followers Β· starred Sep 2026
125 followers Β· starred Jan 2026
bzfs is a CLI built for highly reliable and scalable near real-time ZFS snapshot replication with minimal operational complexity. It reliably replicates ZFS datasets in parallel using zfs send/receive and ssh, and can operate at sub-second intervals across large fleets of hosts for safe DR/HA.
Python
247
2,615 commits
updated Oct 18, 2026
bzfs is a CLI built for highly reliable and scalable near real-time ZFS snapshot replication with minimal operational complexity. It reliably replicates ZFS datasets in parallel using zfs send/receive and ssh, and can operate at sub-second intervals across large fleets of hosts. It safely supports disaster recovery and high availability (DR/HA), scale-out deployments, and protection against data loss or ransomware. It handles the many edge cases that you will eventually run into over the course of your deployment.
On the first run, bzfs replicates the source dataset and all its snapshots to the destination. On subsequent runs, it sends only changes since the previous run by incrementally replicating snapshots created on the source after that run. Source snapshots older than the most recent common snapshot on the destination are skipped automatically.
Unless bzfs is told to create snapshots or bookmarks on the source, it treats the source as read-only. With --dryrun, it
also treats the destination as read-only. In normal operation, the destination is append-only. Optional flags can delete
destination snapshots and datasets if you want to manage storage space consumption, reconcile divergence, restore production
from backup, or resync backup from production.
bzfs supports include/exclude filters that can be combined to choose which datasets, snapshots, and properties to create, replicate, delete, or compare.
A common setup uses scheduled cron jobs: one to create and prune source snapshots, one to prune destination snapshots, and
one to replicate recently created snapshots from source to destination. Schedules can run every N milliseconds, seconds,
minutes, hours, days, weeks, months, or years.
Snapshot creation, replication, pruning, monitoring, and comparison work with snapshots in any naming format, including snapshots created by third-party tools or by manual zfs snapshot commands. These functions can also be used independently.
The source can push to the destination, and the destination can pull from the source. bzfs runs on the initiator host, which
can be the source host (push mode), destination host (pull mode), same host (local mode, no network, no ssh), or a
third-party host that can SSH into source and destination (pull-push mode). In pull-push mode, the source zfs send stream
is relayed by the initiator to the destination zfs receive, without storing anything locally. For bulk data transfers,
remote-to-remote mode (--r2r=pull or --r2r=push) can instead transfer the stream directly between source and destination
to avoid making the initiator a bandwidth bottleneck. In this mode, bzfs does not need to be installed on source or
destination; only the zfs CLI is required there. bzfs can run as root or as a non-root user via sudo or delegated zfs allow permissions.
bzfs is written in Python and continuously tested with unit and integration tests on old and new ZFS versions, on multiple Linux and FreeBSD versions, and on all Python versions >= 3.9 (including latest stable, currently python-3.15).
bzfs is a stand-alone program with zero required dependencies. It is meant to run in restricted barebones server environments. No external Python packages are required; indeed no Python package management at all is required. You can symlink the program wherever you like, such as /usr/local/bin, and run it like a shell script or binary executable.
bzfs replicates snapshots for multiple datasets in parallel. It also deletes (or monitors or compares) snapshots of multiple datasets in parallel. Atomic snapshots can be created as often as every N milliseconds.
Replication progress (e.g. throughput and ETA) is shown aggregated across parallel replication tasks. Example console status line:
2025-01-17 01:23:04 [I] zfs sent 41.7 GiB 0:00:46 [963 MiB/s] [907 MiB/s] 80% ETA 0:00:04 ETA 01:23:08
bzfs uses streaming algorithms to process millions of datasets with low memory usage and low latency. It handles replication policies with multiple sources and multiple destinations per source.
Optionally, bzfs applies bandwidth rate limiting and progress monitoring (via pv) during zfs send/receive transfers. Over
the network, it can insert lightweight compression (via zstd) and buffering (via mbuffer) between endpoints. If one of
these tools is not installed, bzfs auto-detects that and continues without that auxiliary feature.
The project also ships with bzfs_jobrunner, a companion program that wraps bzfs for periodic
snapshot creation, replication, pruning, and monitoring across N source hosts and M destination hosts, using one shared
fleet-wide jobconfig script.
Typical use cases include geo-replicated backup where each destination host is in a different region and receives replicas from the same set of source hosts, low-latency replication from a primary to a secondary or to M read replicas, and backups to removable drives.
$ bzfs tank1/foo/bar dummy --recursive --skip-replication --create-src-snapshots \
--create-src-snapshots-plan "{'test':{'':{'adhoc':1}}}"
$ zfs list -t snapshot tank1/foo/bar
tank1/foo/bar@test_2024-11-06_08:30:05_adhoc
cron job:$ bzfs tank1/foo/bar dummy --recursive --skip-replication --create-src-snapshots \
--create-src-snapshots-plan \
"{'prod':{'us-west':{'minutely':40,'hourly':36,'daily':31}}}"
$ zfs list -t snapshot tank1/foo/bar
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_daily
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_hourly
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_minutely
Note: A periodic snapshot is created if it is due per the schedule indicated by its suffix (e.g. _daily or _hourly or
_minutely or _2secondly or _100millisecondly), or if the --create-src-snapshots-even-if-not-due flag is specified, or
if the most recent scheduled snapshot is somehow missing. In the latter case bzfs immediately creates a snapshot (named with
the current time, not backdated to the missed time), and then resumes the original schedule. If the suffix is _adhoc or not
a known period then a snapshot is considered non-periodic and is thus created immediately regardless of the creation time of
any existing snapshot.
$ bzfs tank1/foo/bar tank2/boo/bar
$ zfs list -t snapshot tank1/foo/bar
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_daily
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_hourly
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_minutely
$ zfs list -t snapshot tank2/boo/bar
tank2/boo/bar@prod_us-west_2024-11-06_08:30:05_daily
tank2/boo/bar@prod_us-west_2024-11-06_08:30:05_hourly
tank2/boo/bar@prod_us-west_2024-11-06_08:30:05_minutely
$ bzfs [email protected]:tank1/foo/bar tank2/boo/bar
$ bzfs tank1/foo/bar [email protected]:tank2/boo/bar
$ bzfs root@host1:tank1/foo/bar root@host2:tank2/boo/bar
--r2r=pull:$ bzfs --r2r=pull root@host1:tank1/foo/bar root@host2:tank2/boo/bar
--r2r=push:$ bzfs --r2r=push root@host1:tank1/foo/bar root@host2:tank2/boo/bar
$ bzfs --recursive tank1/foo/bar tank2/boo/bar
$ zfs list -t snapshot -r tank1/foo/bar
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_daily
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_hourly
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_minutely
tank1/foo/bar/baz@prod_us-west_2024-11-06_08:40:00_daily
tank1/foo/bar/baz@prod_us-west_2024-11-06_08:40:00_hourly
tank1/foo/bar/baz@prod_us-west_2024-11-06_08:40:00_minutely
$ zfs list -t snapshot -r tank2/boo/bar
tank2/boo/bar@prod_us-west_2024-11-06_08:30:05_daily
tank2/boo/bar@prod_us-west_2024-11-06_08:30:05_hourly
tank2/boo/bar@prod_us-west_2024-11-06_08:30:05_minutely
tank2/boo/bar/baz@prod_us-west_2024-11-06_08:40:00_daily
tank2/boo/bar/baz@prod_us-west_2024-11-06_08:40:00_hourly
tank2/boo/bar/baz@prod_us-west_2024-11-06_08:40:00_minutely
$ bzfs tank1/foo/bar tank2/boo/bar --recursive --include-snapshot-plan \
"{'prod':{'us-west':{'minutely':40,'hourly':36,'daily':31}}}"
Note: The example above compares the specified times against the standard ZFS 'creation' time property of the snapshots (which is a UTC Unix time in integer seconds), rather than against a timestamp that may be part of the snapshot name.
$ bzfs dummy tank2/boo/bar --dryrun --recursive --skip-replication \
--delete-dst-snapshots --delete-dst-snapshots-except-plan \
"{'prod':{'us-west':{'secondly':40,'minutely':40,'hourly':36,'daily':31,'weekly':12,'monthly':18,'yearly':5}}}"
Note: This also prints how many GB of disk space in total would be freed if the command were to be run for real without the --dryrun flag.
$ bzfs tank1/foo/bar tank2/boo/bar --skip-replication --compare-snapshot-lists \
--recursive --include-snapshot-regex '.*_(hourly|daily)' \
--include-snapshot-times-and-ranks '7 days ago..4 hours ago' --exclude-dataset-regex \
'(.*/)?tmp.*'
If the resulting TSV output file contains zero lines starting with the prefix 'src' and zero lines starting with the prefix 'dst' then no source snapshots are missing on the destination, and no destination snapshots are missing on the source, indicating that the periodic replication and pruning jobs perform as expected. The TSV output is sorted by dataset, and by ZFS creation time within each dataset - the first and last line prefixed with 'all' contains the metadata of the oldest and latest common snapshot, respectively. The --compare-snapshot-lists option also directly logs various summary stats, such as the metadata of the latest common snapshot, latest snapshots and oldest snapshots, as well as the time diff between the latest common snapshot and latest snapshot only in src (and only in dst), as well as how many src snapshots and how many GB of data are missing on dst, etc.
prod_us-west_<timestamp>_hourly is more than
30 minutes late (i.e. more than 30+60=90 minutes old) [warning], or more than 300 minutes late (i.e. more than 300+60=360
minutes old) [critical]. Analog for minutely and daily snapshots:$ bzfs tank1/foo/bar tank2/boo/bar --recursive --skip-replication -v --monitor-snapshots \
"{'prod':{'us-west':{'minutely':{'latest':{'warning':'30seconds','critical':'300seconds'}},'hourly':{'latest':{'warning':'30minutes','critical':'300minutes'}},'daily':{'latest':{'warning':'4hours','critical':'8hours'}}}}}"
$ bzfs tank1/foo/bar tank2/boo/bar --dryrun --recursive --force --delete-dst-datasets \
--delete-dst-snapshots
# Ubuntu / Debian:
sudo apt-get -y install zfsutils-linux python3 # ensure zfs and python are installed
sudo apt-get -y install zstd pv mbuffer # auxiliary helpers are optional
# RHEL / EL, AlmaLinux, Rocky:
sudo dnf -y install epel-release # auxiliary helpers are optional
sudo dnf -y install zstd pv mbuffer --enablerepo=epel # auxiliary helpers are optional
# FreeBSD:
pkg install -y python3 bash
pkg install -y sudo zstd pv mbuffer # auxiliary helpers are optional
# Install from git repo:
git clone https://github.com/whoschek/bzfs.git
cd bzfs/bzfs_main
./bzfs --help # Run the CLI
./bzfs_jobrunner --help
sudo ln -sf $(pwd)/bzfs /usr/local/bin/bzfs # Optional system installation
sudo ln -sf $(pwd)/bzfs_jobrunner /usr/local/bin/bzfs_jobrunner # Optional system installation
# Alternatively, install a release via pip:
python3 -m pip install bzfs
python3 -m pip install --upgrade bzfs # To upgrade an existing bzfs installation
bzfs --help # Run the CLI
bzfs_jobrunner --help
# Alternatively, setup the environment for software development:
git clone https://github.com/whoschek/bzfs.git
cd bzfs
python3 -m venv .venv # Create a Python virtual environment
. .venv/bin/activate # Activate the virtual environment
. ./preinstall_dev.sh # Prepare install
python3 -m pip install --require-hashes -r requirements-dev.txt # Install all development dependencies
python3 -m pip install -e '.[dev]' --no-deps # Make source edits take effect without reinstalling
pre-commit install --install-hooks # Set up linters/formatters to run on every commit
pre-commit run --all-files # Manually run linters/formatters
--dryrun, so they can be inspected and replayed manually.--dryrun mode, prints what would be deleted, plus estimated reclaimable disk space.sshd.zfs receive -s and zfs send -t).bzfs processes can run in parallel.snapshots_changed to reduce latency by avoiding unnecessary
zfs list -t snapshot calls.zfs send -R -s aka sending a recursive ZFS stream package.Results of continuous integration test runs on a matrix of various old and new versions of ZFS/Python/Linux/FreeBSD are here, as generated by this script. The script also demonstrates functioning installation steps on Ubuntu and FreeBSD, etc. It also generates code coverage reports which are published here.
For reproducible local testing, consider using lima_vm.sh.
On macOS or Linux, this script uses Lima to locally create a guest Ubuntu or AlmaLinux VM, then installs ZFS and runs
the bzfs test suite inside of that VM.
Also consider running the lima_testbed.sh script that uses this to create/delete a
local Lima testbed with N source VMs and M destination VMs for testing, with ZFS and VM-to-VM SSH connectivity working
out of the box. All default settings work out of the box.
Then consider running the example replication jobconfig script bzfs_job_testbed.py
on the testbed, which also works out of the box.
# Check prerequisites
zfs --version # verify ZFS is installed
python3 --version # verify Python 3.9 or newer is installed
sudo -n ls # verify non-interactive sudo works
# If sshd uses a non-standard port
# export bzfs_test_ssh_port=2222
# Choose test scope
# export bzfs_test_mode=unit # unit tests only (takes < 5 seconds)
export bzfs_test_mode=smoke # small, quick integration test subset (takes < 1 minute)
# export bzfs_test_mode=functional # most integration tests but only in a single local config (takes ~3 minutes)
# unset bzfs_test_mode # all tests (takes ~15 minutes)
export bzfs_test_no_run_quietly=true # print more than just a progress bar
# Confirm passwordless ssh works on connecting to localhost loopback address
# You should not get prompted for a password
ssh -p ${bzfs_test_ssh_port:-22} 127.0.0.1 echo hello
# Ensure tools are on remote PATH
ssh -p ${bzfs_test_ssh_port:-22} 127.0.0.1 zfs --version
ssh -p ${bzfs_test_ssh_port:-22} 127.0.0.1 zpool --version
ssh -p ${bzfs_test_ssh_port:-22} 127.0.0.1 pv --version # enables progress reporting
ssh -p ${bzfs_test_ssh_port:-22} 127.0.0.1 zstd --version # enables compression-on-the-wire
ssh -p ${bzfs_test_ssh_port:-22} 127.0.0.1 mbuffer --version # enables buffering
# Run the tests
./test.sh # when cloned from git
# or
bzfs-test # when installed via pip
Actions tab in your fork and enable GitHub Actions.All workflows -> Tests in the left sidebar.Run workflow dropdown menu on the right. The UI looks like
this screenshot.test_ubuntu_26_04 or test_ubuntu_24_04 or test_freebsd_15_1 or test_almalinux_10_zfs_2_4_hpnssh) or
choose Run all jobs to run the test suite on the full matrix of all supported platforms.usage: bzfs [-h]
[--recursive]
[--include-dataset DATASET [DATASET ...]]
[--exclude-dataset DATASET [DATASET ...]]
[--include-dataset-regex REGEX [REGEX ...]]
[--exclude-dataset-regex REGEX [REGEX ...]]
[--exclude-dataset-property STRING]
[--include-snapshot-regex REGEX [REGEX ...]]
[--exclude-snapshot-regex REGEX [REGEX ...]]
[--include-snapshot-times-and-ranks TIMERANGE [RANKRANGE ...]]
[--include-snapshot-plan DICT_STRING]
[--new-snapshot-filter-group]
[--create-src-snapshots]
[--create-src-snapshots-plan DICT_STRING]
[--create-src-snapshots-plan-attributes DICT_STRING]
[--create-src-snapshots-timeformat STRFTIME_SPEC]
[--create-src-snapshots-timezone TZ_SPEC]
[--create-src-snapshots-even-if-not-due]
[--zfs-send-program-opts STRING]
[--zfs-recv-program-opts STRING]
[--zfs-recv-program-opt STRING]
[--preserve-properties STRING [STRING ...]]
[--force-rollback-to-latest-snapshot]
[--force-rollback-to-latest-common-snapshot]
[--force]
[--force-destroy-dependents]
[--force-unmount]
[--force-once]
[--skip-parent]
[--skip-missing-snapshots [{fail,dataset,continue}]]
[--retries INT]
[--retry-min-sleep-secs FLOAT]
[--retry-initial-max-sleep-secs FLOAT]
[--retry-max-sleep-secs FLOAT]
[--retry-max-elapsed-secs FLOAT]
[--skip-on-error {fail,tree,dataset}]
[--skip-replication]
[--delete-dst-datasets]
[--delete-dst-snapshots [{snapshots,bookmarks}]]
[--delete-dst-snapshots-no-crosscheck]
[--delete-dst-snapshots-except]
[--delete-dst-snapshots-except-plan DICT_STRING]
[--delete-empty-dst-datasets [{snapshots,snapshots+bookmarks}]]
[--monitor-snapshots DICT_STRING]
[--monitor-snapshots-dont-warn]
[--monitor-snapshots-dont-crit]
[--compare-snapshot-lists [{src,dst,all,src+dst,src+all,dst+all,src+dst+all}]]
[--cache-snapshots]
[--dryrun [{recv,send}]]
[--verbose]
[--quiet]
[--no-privilege-elevation]
[--no-stream]
[--no-resume-recv]
[--create-bookmarks {all,hourly,minutely,secondly,none}]
[--no-use-bookmark]
[--ssh-cipher STRING]
[--ssh-src-user STRING]
[--ssh-dst-user STRING]
[--ssh-src-host STRING]
[--ssh-dst-host STRING]
[--ssh-src-port INT]
[--ssh-dst-port INT]
[--ssh-src-config-file FILE]
[--ssh-dst-config-file FILE]
[--ssh-control-persist-secs INT]
[--threads INT[%]]
[--max-concurrent-ssh-sessions-per-tcp-connection INT]
[--r2r {off,pull,push}]
[--bwlimit STRING]
[--no-estimate-send-size]
[--compression-program {zstd,lz4,pzstd,pigz,gzip,-}]
[--compression-program-opts STRING]
[--mbuffer-program {mbuffer,-}]
[--mbuffer-program-opts STRING]
[--ps-program {ps,-}]
[--pv-program {pv,-}]
[--pv-program-opts STRING]
[--shell-program {sh,-}]
[--ssh-program {ssh,hpnssh,-}]
[--sudo-program {sudo,doas,-}]
[--zpool-program {zpool,-}]
[--log-dir DIR]
[--log-file-prefix STRING]
[--log-file-infix STRING]
[--log-file-suffix STRING]
[--log-subdir {daily,hourly,minutely}]
[--log-syslog-address STRING]
[--log-syslog-socktype {UDP,TCP}]
[--log-syslog-facility INT]
[--log-syslog-prefix STRING]
[--log-syslog-level {CRITICAL,ERROR,WARN,INFO,DEBUG,TRACE}]
[--include-envvar-regex REGEX [REGEX ...]]
[--exclude-envvar-regex REGEX [REGEX ...]]
[--yearly_year INT]
[--yearly_month INT]
[--yearly_monthday INT]
[--yearly_hour INT]
[--yearly_minute INT]
[--yearly_second INT]
[--monthly_month INT]
[--monthly_monthday INT]
[--monthly_hour INT]
[--monthly_minute INT]
[--monthly_second INT]
[--weekly_weekday INT]
[--weekly_hour INT]
[--weekly_minute INT]
[--weekly_second INT]
[--daily_hour INT]
[--daily_minute INT]
[--daily_second INT]
[--hourly_minute INT]
[--hourly_second INT]
[--minutely_second INT]
[--secondly_millisecond INT]
[--millisecondly_microsecond INT]
[--zfs-recv-o-targets {full,incremental,full+incremental}]
[--zfs-recv-o-sources STRING]
[--zfs-recv-o-include-regex [REGEX ...]]
[--zfs-recv-o-exclude-regex REGEX [REGEX ...]]
[--zfs-recv-x-targets {full,incremental,full+incremental}]
[--zfs-recv-x-sources STRING]
[--zfs-recv-x-include-regex [REGEX ...]]
[--zfs-recv-x-exclude-regex REGEX [REGEX ...]]
[--version]
SRC_DATASET DST_DATASET [SRC_DATASET DST_DATASET ...]
SRC_DATASET DST_DATASET [SRC_DATASET DST_DATASET ...] π
SRC_DATASET: Source ZFS dataset (and its descendants) that will be replicated. Can be a ZFS filesystem or ZFS volume.
Format is [[user@]host:]dataset. The host name can also be an IPv4 address (or an IPv6 address where each ':' colon
character must be replaced with a '|' pipe character for disambiguation). If the host name is '-', the dataset will be on
the local host, and the corresponding SSH leg will be omitted. The same is true if the host is omitted and the dataset does
not contain a ':' colon at the same time. Local dataset examples: tank1/foo/bar, tank1, -:tank1/foo/bar:baz:boo
Remote dataset examples: host:tank1/foo/bar, host.example.com:tank1/foo/bar, root@host:tank,
[email protected]:tank1/foo/bar, [email protected]:tank1/foo/bar:baz:boo, user@||1:tank1/foo/bar:baz:boo. The first
component of the ZFS dataset name is the ZFS pool name, here tank1. If the option starts with a + prefix then dataset
names are read from the UTF-8 text file given after the + prefix, with each line in the file containing a SRC_DATASET and
a DST_DATASET, separated by a tab character. The basename must contain the substring 'bzfs_argument_file'. Example:
+root_dataset_names_bzfs_argument_file.txt, +/path/to/root_dataset_names_bzfs_argument_file.txt
DST_DATASET: Destination ZFS dataset for replication and deletion. Has same naming format as SRC_DATASET. During replication, destination datasets that do not yet exist are created as necessary, along with their parent and ancestors.
Performance Note: bzfs automatically replicates multiple datasets in parallel. It replicates snapshots in parallel across datasets and serially within a dataset. All child datasets of a dataset may be processed in parallel. For consistency, processing of a dataset only starts after processing of all its ancestor datasets has completed. Further, when a thread is ready to start processing another dataset, it chooses the next dataset wrt. lexicographical sort order from the datasets that are currently available for start of processing. Initially, only the roots of the selected dataset subtrees are available for start of processing. The degree of parallelism is configurable with the --threads option (see below).
-h, --help π
--recursive, -r π
--include-dataset DATASET [DATASET ...] π
During snapshot creation, replication, deletion and comparison, select any ZFS dataset (and its descendants) that is contained within SRC_DATASET (DST_DATASET in case of deletion) if its dataset name is one of the given include dataset names but none of the exclude dataset names. If a dataset is excluded its descendants are automatically excluded too, and this decision is never reconsidered even for the descendants because exclude takes precedence over include.
A dataset name is absolute if the specified dataset is prefixed by /, e.g. /tank/baz/tmp. Otherwise the dataset name is
relative wrt. source and destination, e.g. baz/tmp if the source is tank.
This option is automatically translated to an --include-dataset-regex (see below) and can be specified multiple times.
If the option starts with a + prefix then dataset names are read from the newline-separated UTF-8 text file given after
the + prefix, one dataset per line inside of the text file. The basename must contain the substring 'bzfs_argument_file'.
Examples: /tank/baz/tmp (absolute), baz/tmp (relative), +dataset_names_bzfs_argument_file.txt,
+/path/to/dataset_names_bzfs_argument_file.txt
--exclude-dataset DATASET [DATASET ...] π
--include-dataset-regex REGEX [REGEX ...] π
During snapshot creation, replication (and deletion) and comparison, select any ZFS dataset (and its descendants) that is
contained within SRC_DATASET (DST_DATASET in case of deletion) if its relative dataset path (e.g. baz/tmp) wrt.
SRC_DATASET (DST_DATASET in case of deletion) matches at least one of the given include regular expressions but none of the
exclude regular expressions. If a dataset is excluded its descendants are automatically excluded too, and this decision is
never reconsidered even for the descendants because exclude takes precedence over include.
This option can be specified multiple times. A leading ! character indicates logical negation, i.e. the regex matches if
the regex with the leading ! character removed does not match.
If the option starts with a + prefix then regex names are read from the newline-separated UTF-8 text file given after the
+ prefix, one regex per line inside of the text file. The basename must contain the substring 'bzfs_argument_file'.
Default: .* (include all datasets).
Examples: baz/tmp, (.*/)?doc[^/]*/(private|confidential).*, !public, +dataset_regexes_bzfs_argument_file.txt,
+/path/to/dataset_regexes_bzfs_argument_file.txt
--exclude-dataset-regex REGEX [REGEX ...] π
(?:.*/)?[Tt][Ee]?[Mm][Pp][-_]?[0-9]*
(exclude tmp datasets). Example: !.* (exclude no dataset)--exclude-dataset-property STRING π
The name of a ZFS dataset user property (optional). If this option is specified, the effective value (potentially inherited) of that user property is read via 'zfs list' for each selected source dataset to determine whether the dataset will be included or excluded, as follows:
a) Value is 'true' or '-' or empty string or the property is missing: Include the dataset.
b) Value is 'false': Exclude the dataset and its descendants.
c) Value is a comma-separated list of host names (no spaces, for example: 'store001,store002'): Include the dataset if the host name of the host executing bzfs is contained in the list, otherwise exclude the dataset and its descendants.
If a dataset is excluded its descendants are automatically excluded too, and the property values of the descendants are ignored because exclude takes precedence over include.
Examples: 'syncoid:sync', 'com.example.eng.project.x:backup'
Note: The use of --exclude-dataset-property is discouraged for most use cases. It is more flexible, more powerful, and more efficient to instead use a combination of --include/exclude-dataset-regex and/or --include/exclude-dataset to achieve the same or better outcome.
--include-snapshot-regex REGEX [REGEX ...] π
During replication, deletion and comparison, select any source ZFS snapshot that has a name (i.e. the part after the '@') that matches at least one of the given include regular expressions but none of the exclude regular expressions. If a snapshot is excluded this decision is never reconsidered because exclude takes precedence over include.
This option can be specified multiple times. A leading ! character indicates logical negation, i.e. the regex matches if
the regex with the leading ! character removed does not match.
Default: .* (include all snapshots). Examples: test_.*, !prod_.*, .*_(hourly|frequent), !.*_(weekly|daily)
Note: All --include/exclude-snapshot-* CLI option groups are combined into a mini filter pipeline. A filter pipeline is executed in the order given on the command line, left to right. For example if --include-snapshot-times-and-ranks (see below) is specified on the command line before --include/exclude-snapshot-regex, then --include-snapshot-times-and-ranks will be applied before --include/exclude-snapshot-regex. The pipeline results would not always be the same if the order were reversed. Order matters.
Note: During replication, bookmarks are always retained aka selected in order to help find common snapshots between source and destination.
--exclude-snapshot-regex REGEX [REGEX ...] π
--include-snapshot-times-and-ranks TIMERANGE [RANKRANGE ...] π
This option takes as input parameters a time range filter and an optional rank range filter. It separately computes the results for each filter and selects the UNION of both results. To instead use a pure rank range filter (no UNION), or a pure time range filter (no UNION), simply use 'notime' aka '0..0' to indicate an empty time range, or omit the rank range, respectively. This option can be specified multiple times.
Replication Example (UNION):
Specify to replicate all daily snapshots created during the last 7 days, and at the same time ensure that the latest 7
daily snapshots (per dataset) are replicated regardless of creation time, like so: --include-snapshot-regex '.*_daily' --include-snapshot-times-and-ranks '7 days ago..anytime' 'latest 7'
Deletion Example (no UNION):
Specify to delete all daily snapshots older than 7 days, but ensure that the latest 7 daily snapshots (per dataset) are
retained regardless of creation time, like so: --include-snapshot-regex '.*_daily' --include-snapshot-times-and-ranks notime 'all except latest 7' --include-snapshot-times-and-ranks 'anytime..7 days ago'
This helps to safely cope with irregular scenarios where no snapshots were created or received within the last 7 days, or where more than 7 daily snapshots were created within the last 7 days. It can also help to avoid accidental pruning of the last snapshot that source and destination have in common.
TIMERANGE:
The ZFS 'creation' time of a snapshot (and bookmark) must fall into this time range in order for the snapshot to be
included. The time range consists of a 'start' time, followed by a '..' separator, followed by an 'end' time. For example
'2024-01-01..2024-04-01', or 'anytime..anytime' aka *..* aka all times, or 'notime' aka '0..0' aka empty time range. Only
snapshots (and bookmarks) in the half-open time range [start, end) are included; other snapshots (and bookmarks) are
excluded. If a snapshot is excluded this decision is never reconsidered because exclude takes precedence over include. Each
of the two specified times can take any of the following forms:
a) anytime aka * wildcard; represents negative or positive infinity.
b) a non-negative integer representing a UTC Unix time in seconds. Example: 1728109805
c) an ISO 8601 datetime string with or without timezone. Examples: '2024-10-05', '2024-10-05T14:48:55', '2024-10-05T14:48:55+02', '2024-10-05T14:48:55-04:30'. If the datetime string does not contain time zone info then it is assumed to be in the local time zone. Timezone string support requires Python β₯ 3.11.
d) a duration that indicates how long ago from the current time, using the following syntax: a non-negative integer, followed by an optional space, followed by a duration unit that is one of 'seconds', 'secs', 'minutes', 'mins', 'hours', 'days', 'weeks', 'months', 'years', followed by an optional space, followed by the word 'ago'. Examples: '0secs ago', '40 mins ago', '36hours ago', '90days ago', '12weeksago'.
Note: This option compares the specified time against the standard ZFS 'creation' time property of the snapshot (which is
a UTC Unix time in integer seconds), rather than against a timestamp that may be part of the snapshot name. You can list
the ZFS creation time of snapshots and bookmarks as follows: zfs list -t snapshot,bookmark -o name,creation -s creation -d 1 $SRC_DATASET (optionally add the -p flag to display UTC Unix time in integer seconds).
Note: During replication, bookmarks are always retained aka selected in order to help find common snapshots between source and destination.
RANKRANGE:
Specifies to include the N (or N%) oldest snapshots or latest snapshots, and exclude all other snapshots (default: include no snapshots). Snapshots are sorted by creation time (actually, by the 'createtxg' ZFS property, which serves the same purpose but is more precise). The rank position of a snapshot is the zero-based integer position of the snapshot within that sorted list. A rank consists of the optional words 'all except' (followed by an optional space), followed by the word 'oldest' or 'latest', followed by a non-negative integer, followed by an optional '%' percent sign. A rank range consists of a lower rank, followed by a '..' separator, followed by a higher rank. If the optional lower rank is missing it is assumed to be 0. Examples:
'oldest 10%' aka 'oldest 0..oldest 10%' (include the oldest 10% of all snapshots)
'latest 10%' aka 'latest 0..latest 10%' (include the latest 10% of all snapshots)
'all except latest 10%' aka 'oldest 90%' aka 'oldest 0..oldest 90%' (include all snapshots except the latest 10% of all snapshots)
'oldest 90' aka 'oldest 0..oldest 90' (include the oldest 90 snapshots)
'latest 90' aka 'latest 0..latest 90' (include the latest 90 snapshots)
'all except oldest 90' aka 'oldest 90..oldest 100%' (include all snapshots except the oldest 90 snapshots)
'all except latest 90' aka 'latest 90..latest 100%' (include all snapshots except the latest 90 snapshots)
'latest 1' aka 'latest 0..latest 1' (include the latest snapshot)
'all except latest 1' aka 'latest 1..latest 100%' (include all snapshots except the latest snapshot)
'oldest 2' aka 'oldest 0..oldest 2' (include the oldest 2 snapshots)
'all except oldest 2' aka 'oldest 2..oldest 100%' (include all snapshots except the oldest 2 snapshots)
'oldest 100%' aka 'oldest 0..oldest 100%' (include all snapshots)
'oldest 0%' aka 'oldest 0..oldest 0%' (include no snapshots)
'oldest 0' aka 'oldest 0..oldest 0' (include no snapshots)
Note for multiple RANKRANGEs: --include-snapshot-times-and-ranks TIMERANGE RANKRANGE1 RANKRANGE2 is equivalent to
--include-snapshot-times-and-ranks TIMERANGE RANKRANGE1 --include-snapshot-times-and-ranks TIMERANGE RANKRANGE2.
Note: Percentage calculations are not based on the number of snapshots contained in the dataset on disk, but rather based on the number of snapshots arriving at the filter. For example, if only two daily snapshots arrive at the filter because a prior filter excludes hourly snapshots, then 'latest 10' will only include these two daily snapshots, and 'latest 50%' will only include one of these two daily snapshots.
Note: During replication, bookmarks are always retained aka selected in order to help find common snapshots between source and destination. Bookmarks do not count towards N or N% wrt. rank.
Note: If a snapshot is excluded this decision is never reconsidered because exclude takes precedence over include.
--include-snapshot-plan DICT_STRING π
Replication periods to be used if replicating snapshots within the selected destination datasets. Has the same format as --create-src-snapshots-plan and --delete-dst-snapshots-except-plan (see below). Snapshots that do not match a period will not be replicated. To avoid unexpected surprises, make sure to carefully specify ALL snapshot names and periods that shall be replicated, in combination with --dryrun.
Example: "{'prod': {'onsite': {'secondly': 40, 'minutely': 40, 'hourly': 36, 'daily': 31, 'weekly': 12, 'monthly': 18, 'yearly': 5}, 'us-west': {'secondly': 0, 'minutely': 0, 'hourly': 36, 'daily': 31, 'weekly': 12, 'monthly': 18, 'yearly': 5}, 'eu-west': {'secondly': 0, 'minutely': 0, 'hourly': 36, 'daily': 31, 'weekly': 12, 'monthly': 18, 'yearly': 5}}, 'test': {'offsite': {'12hourly': 42, 'weekly': 12}, 'onsite': {'100millisecondly': 42}}}". This example will, for the
organization 'prod' and the intended logical target 'onsite', replicate secondly snapshots that were created less than 40
seconds ago, yet replicate the latest 40 secondly snapshots regardless of creation time. Analog for the latest 40 minutely
snapshots, latest 36 hourly snapshots, etc. Note: A zero within a period (e.g. 'hourly': 0) indicates that no snapshots
shall be replicated for the given period.
Note: --include-snapshot-plan is a convenience option that auto-generates a series of the following other options: --new-snapshot-filter-group, --include-snapshot-regex, --include-snapshot-times-and-ranks
--new-snapshot-filter-group π
Starts a new snapshot filter group containing separate --{include|exclude}-snapshot-* filter options. The program separately computes the results for each filter group and selects the UNION of all results. This option can be specified multiple times and serves as a separator between groups. Example:
Delete all minutely snapshots older than 40 minutes, but ensure that the latest 40 minutely snapshots (per dataset) are
retained regardless of creation time. Additionally, delete all hourly snapshots older than 36 hours, but ensure that the
latest 36 hourly snapshots (per dataset) are retained regardless of creation time. Additionally, delete all daily snapshots
older than 31 days, but ensure that the latest 31 daily snapshots (per dataset) are retained regardless of creation time:
bzfs dummy tank2/boo/bar --dryrun --recursive --skip-replication --delete-dst-snapshots --include-snapshot-regex '.*_minutely' --include-snapshot-times-and-ranks notime 'all except latest 40' --include-snapshot-times-and-ranks 'anytime..40 minutes ago' --new-snapshot-filter-group --include-snapshot-regex '.*_hourly' --include-snapshot-times-and-ranks notime 'all except latest 36' --include-snapshot-times-and-ranks 'anytime..36 hours ago' --new-snapshot-filter-group --include-snapshot-regex '.*_daily' --include-snapshot-times-and-ranks notime 'all except latest 31' --include-snapshot-times-and-ranks 'anytime..31 days ago'
--create-src-snapshots π
Do nothing if the --create-src-snapshots flag is missing. Otherwise, before the replication step (see below), atomically create new snapshots of the source datasets selected via --{include|exclude}-dataset* policy. The names of the snapshots can be configured via --create-src-snapshots-* suboptions (see below). To create snapshots only, without any other processing such as replication, etc, consider using this flag together with the --skip-replication flag.
A periodic snapshot is created if it is due per the schedule indicated by --create-src-snapshots-plan (for example '_daily' or '_hourly' or _'10minutely' or '_2secondly' or '_100millisecondly'), or if the --create-src-snapshots-even-if-not-due flag is specified, or if the most recent scheduled snapshot is somehow missing. In the latter case bzfs immediately creates a snapshot (tagged with the current time, not backdated to the missed time), and then resumes the original schedule.
If the snapshot suffix is '_adhoc' or not a known period then a snapshot is considered non-periodic and is thus created immediately regardless of the creation time of any existing snapshot.
The implementation attempts to fit as many datasets as possible into a single (atomic) 'zfs snapshot' command line, using
lexicographical sort order, and using 'zfs snapshot -r' to the extent that this is compatible with the actual results of
the schedule and the actual results of the --{include|exclude}-dataset* pruning policy. The snapshots of all datasets that
fit within the same single 'zfs snapshot' CLI invocation will be taken within the same ZFS transaction group, and
correspondingly have identical 'createtxg' ZFS property (but not necessarily identical 'creation' ZFS time property as ZFS
actually provides no such guarantee), and thus be consistent. Dataset names that can't fit into a single command line are
spread over multiple command line invocations, respecting the limits that the operating system places on the maximum length
of a single command line, per getconf ARG_MAX.
Note: All bzfs functions including snapshot creation, replication, deletion, monitoring, comparison, etc. happily work with any snapshots in any format, even created or managed by third party ZFS snapshot management tools, including manual zfs snapshot/destroy.
--create-src-snapshots-plan DICT_STRING π
Creation periods that specify a schedule for when new snapshots shall be created on src within the selected datasets. Has the same format as --delete-dst-snapshots-except-plan.
Example: "{'prod': {'onsite': {'secondly': 40, 'minutely': 40, 'hourly': 36, 'daily': 31, 'weekly': 12, 'monthly': 18, 'yearly': 5}, 'us-west': {'secondly': 0, 'minutely': 0, 'hourly': 36, 'daily': 31, 'weekly': 12, 'monthly': 18, 'yearly': 5}, 'eu-west': {'secondly': 0, 'minutely': 0, 'hourly': 36, 'daily': 31, 'weekly': 12, 'monthly': 18, 'yearly': 5}}, 'test': {'offsite': {'12hourly': 42, 'weekly': 12}, 'onsite': {'100millisecondly': 42}}}". This example will, for the
organization 'prod' and the intended logical target 'onsite', create 'secondly' snapshots every second, 'minutely'
snapshots every minute, hourly snapshots every hour, and so on. It will also create snapshots for the targets 'us-west' and
'eu-west' within the 'prod' organization. In addition, it will create snapshots every 12 hours and every week for the
'test' organization, and name them as being intended for the 'offsite' replication target. Analog for snapshots that are
taken every 100 milliseconds within the 'test' organization.
The example creates ZFS snapshots with names like prod_onsite_<timestamp>_secondly, prod_onsite_<timestamp>_minutely,
prod_us-west_<timestamp>_hourly, prod_us-west_<timestamp>_daily, prod_eu-west_<timestamp>_hourly,
prod_eu-west_<timestamp>_daily, test_offsite_<timestamp>_12hourly, test_offsite_<timestamp>_weekly, and so on.
Note: A period name that is missing indicates that no snapshots shall be created for the given period.
The period name can contain an optional positive integer immediately preceding the time period unit, for example
_2secondly or _10minutely or _100millisecondly to indicate that snapshots are taken every 2 seconds, or every 10
minutes, or every 100 milliseconds, respectively.
--create-src-snapshots-plan-attributes DICT_STRING π
Default is "{}". Optional additional parameters that specify how to create snapshots on a per-period basis. Has the same
format as --create-src-snapshots-plan except nested values are dicts instead of integers.
Example: "{'prod': {'onsite': {'secondly': {'written_threshold': 1}}}}". This example will, for the organization 'prod'
and the intended logical target 'onsite', create a 'secondly' snapshot every second if ZFS has written at least 1 byte to
the dataset since its latest snapshot per the standard 'written' ZFS dataset property. The default for each period is
"{'written_threshold': 0}", i.e. take a snapshot even if nothing has been written. See
https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html#written
--create-src-snapshots-timeformat STRFTIME_SPEC π
Default is %Y-%m-%d_%H:%M:%S. For the strftime format, see
https://docs.python.org/3.11/library/datetime.html#strftime-strptime-behavior. Examples: %Y-%m-%d_%H:%M:%S.%f (adds
microsecond resolution), %Y-%m-%d_%H:%M:%S%z (adds timezone offset), %Y-%m-%dT%H-%M-%S (no colons).
The name of the snapshot created on the src is $org_$target_strftime(--create-src-snapshots-time*)_$period. Example:
tank/foo@prod_us-west_2024-09-03_12:26:15_daily
--create-src-snapshots-timezone TZ_SPEC π
Default is the local timezone of the system running bzfs. When creating a new snapshot on the source, fetch the current time in the specified timezone, and feed that time, and the value of --create-src-snapshots-timeformat, into the standard strftime() function to generate the timestamp portion of the snapshot name. The TZ_SPEC input parameter is of the form 'UTC' or '+HHMM' or '-HHMM' for fixed UTC offsets, or an IANA TZ identifier for auto-adjustment to daylight savings time, or the empty string to use the local timezone, for example '', 'UTC', '+0000', '+0530', '-0400', 'America/Los_Angeles', 'Europe/Vienna'. For a list of valid IANA TZ identifiers see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
To change the timezone not only for snapshot name creation, but in all respects for the entire program, use the standard
'TZ' Unix environment variable, like so: export TZ=UTC.
--create-src-snapshots-even-if-not-due π
--zfs-send-program-opts STRING π
Parameters to fine-tune 'zfs send' behaviour (optional); will be passed into 'zfs send' CLI. The value is split on runs of
one or more whitespace characters. Default is '--raw --compressed'. To run zfs send without options, specify the empty
string: --zfs-send-program-opts=''. See https://openzfs.github.io/openzfs-docs/man/master/8/zfs-send.8.html and
https://github.com/openzfs/zfs/issues/13024
If these options request a different effective raw or non-raw mode from the one recorded in an existing ZFS receive resume token, bzfs aborts the token (clears the incomplete receive state) and then proceeds as usual. ZFS may be unable to change the mode for an existing destination dataset. In that case, restore the previous send options.
--zfs-recv-program-opts STRING π
zfs receive without options, specify the empty string:
--zfs-recv-program-opts=''. Example: '-u -o canmount=noauto -o readonly=on -x keylocation -x keyformat -x encryption'.
See https://openzfs.github.io/openzfs-docs/man/master/8/zfs-receive.8.html and
https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html--zfs-recv-program-opt STRING π
--zfs-recv-program-opt=-o --zfs-recv-program-opt='org.zfsbootmenu:commandline=ro debug zswap.enabled=1'--preserve-properties STRING [STRING ...] π
On replication, preserve the current value of ZFS properties with the given names on the destination datasets. The destination ignores the property value it zfs receive's from the source if the property name matches one of the given blacklist values. This prevents a compromised or untrusted source from overwriting security-critical properties on the destination. The default is to preserve none, i.e. an empty blacklist.
Example blacklist that protects against dangerous overwrites: mountpoint overlay sharenfs sharesmb exec setuid devices encryption keyformat keylocation
See https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html and https://openzfs.github.io/openzfs-docs/man/master/8/zfs-receive.8.html#x
Note: --preserve-properties uses the 'zfs recv -x' option and thus requires either OpenZFS β₯ 2.2.0 (see https://github.com/openzfs/zfs/commit/b0269cd8ced242e66afc4fa856d62be29bb5a4ff), or that 'zfs send --props' is not used.
--force-rollback-to-latest-snapshot π
--force-rollback-to-latest-common-snapshot π
--force π
Same as --force-rollback-to-latest-common-snapshot (see above), except that additionally, if no common snapshot exists, then delete all destination snapshots before starting replication, and proceed without aborting. Without the --force* flags, the destination dataset is treated as append-only, hence no destination snapshot that already exists is deleted, and instead the operation is aborted with an error when encountering a conflicting snapshot.
Analogy: --force-rollback-to-latest-snapshot is a tiny hammer, whereas --force-rollback-to-latest-common-snapshot is a medium sized hammer, --force is a large hammer, and --force-destroy-dependents is a very large hammer. Consider using the smallest hammer that can fix the problem. No hammer is ever used by default.
--force-destroy-dependents π
--force-unmount π
--force-once, --f1 π
--skip-parent π
bzfs --recursive --skip-parent src dst is akin to Unix cp -r src/* dst/ whereas bzfs --recursive --skip-parent --skip-replication --delete-dst-datasets dummy dst is akin to Unix rm -r dst/*--skip-missing-snapshots [{fail,dataset,continue}] π
During replication, handle source datasets that select no snapshots (and no relevant bookmarks) as follows:
a) 'fail': Abort with an error.
b) 'dataset' (default): Skip the source dataset with a warning. Skip descendant datasets if --recursive and destination dataset does not exist or is inconsistent. Otherwise skip to the next dataset.
c) 'continue': Continue processing the next dataset, including descendants. If an inconsistent destination dataset prevents processing selected descendants, fail with an error; --skip-on-error determines whether processing continues elsewhere. Eventually create empty destination dataset and ancestors if they do not yet exist and source dataset has at least one descendant that selects at least one snapshot.
--retries INT π
--retry-min-sleep-secs FLOAT π
--retry-initial-max-sleep-secs FLOAT π
--retry-max-sleep-secs FLOAT π
--retry-max-elapsed-secs FLOAT π
--skip-on-error {fail,tree,dataset} π
During replication and deletion, if an error is not retryable, or --retries has been exhausted, or --skip-missing-snapshots raises an error, proceed as follows:
a) 'fail': Abort the program with an error. This mode is ideal for testing, clear error reporting, and situations where consistency trumps availability.
b) 'tree': Log the error, skip the dataset tree rooted at the dataset for which the error occurred, and continue processing the next (sibling) dataset tree. Example: Assume datasets tank/user1/foo and tank/user2/bar and an error occurs while processing tank/user1. In this case processing skips tank/user1/foo and proceeds with tank/user2.
c) 'dataset' (default): Same as 'tree' except if the destination dataset already exists and is not inconsistent, skip to the next dataset instead.
Example: Assume datasets tank/user1/foo and tank/user2/bar and an error occurs while processing tank/user1. In this case processing skips tank/user1 and proceeds with tank/user1/foo if the destination already contains tank/user1 and that dataset is not inconsistent. Otherwise processing continues with tank/user2. This mode is for production use cases that require timely forward progress even in the presence of partial failures. For example, assume the job is to backup the home directories or virtual machines of thousands of users across an organization. Even if replication of some of the datasets for some users fails due too conflicts, busy datasets, etc, the replication job will continue for the remaining datasets and the remaining users.
--skip-replication π
--delete-dst-datasets π
Do nothing if the --delete-dst-datasets option is missing. Otherwise, after successful replication step, if any, delete existing destination datasets that are selected via --{include|exclude}-dataset* policy yet do not exist within SRC_DATASET (which can be an empty dataset, such as the hardcoded virtual dataset named 'dummy'!). Do not recurse without --recursive. With --recursive, never delete non-selected dataset subtrees or their ancestors.
For example, if the destination contains datasets h1,h2,h3,d1 whereas source only contains h3, and the include/exclude policy selects h1,h2,h3,d1, then delete datasets h1,h2,d1 on the destination to make it 'the same'. On the other hand, if the include/exclude policy only selects h1,h2,h3 then only delete datasets h1,h2 on the destination to make it 'the same'.
Example to delete all tmp datasets within tank2/boo/bar: bzfs dummy tank2/boo/bar --dryrun --skip-replication --recursive --delete-dst-datasets --include-dataset-regex '(.*/)?tmp.*' --exclude-dataset-regex '!.*'
--delete-dst-snapshots [{snapshots,bookmarks}] π
Do nothing if the --delete-dst-snapshots option is missing. Otherwise, after successful replication, and successful --delete-dst-datasets step, if any, delete existing destination snapshots whose GUID does not exist within the source dataset (which can be an empty dummy dataset!) if the destination snapshots are selected by the --include/exclude-snapshot-* policy, and the destination dataset is selected via --{include|exclude}-dataset* policy. Does not recurse without --recursive.
For example, if the destination dataset contains snapshots h1,h2,h3,d1 (h=hourly, d=daily) whereas the source dataset only contains snapshot h3, and the include/exclude policy selects h1,h2,h3,d1, then delete snapshots h1,h2,d1 on the destination dataset to make it 'the same'. On the other hand, if the include/exclude policy only selects snapshots h1,h2,h3 then only delete snapshots h1,h2 on the destination dataset to make it 'the same'.
Note: To delete snapshots regardless, consider using --delete-dst-snapshots in combination with a source that is an empty
dataset, such as the hardcoded virtual dataset named 'dummy', like so: bzfs dummy tank2/boo/bar --dryrun --skip-replication --delete-dst-snapshots --include-snapshot-regex '.*_daily' --recursive
Note: Use --delete-dst-snapshots=bookmarks to delete non-temporary bookmarks instead of snapshots, in which case no snapshots are selected and the --{include|exclude}-snapshot-* filter options treat bookmarks as snapshots wrt. selecting.
Note: Does not attempt to delete snapshots that carry a zfs hold; instead auto-skips them without failing.
Performance Note: --delete-dst-snapshots operates on multiple datasets in parallel (and serially within a dataset), using the same dataset order as bzfs replication. The degree of parallelism is configurable with the --threads option (see below).
--delete-dst-snapshots-no-crosscheck π
--delete-dst-snapshots-except π
This flag indicates that the --include/exclude-snapshot-* options shall have inverted semantics for the --delete-dst-snapshots option, thus deleting all snapshots except for the selected snapshots (within the specified datasets), instead of deleting all selected snapshots (within the specified datasets). In other words, this flag enables to specify which snapshots to retain instead of which snapshots to delete.
Synchronization vs. Backup: When a real (non-dummy) source dataset is specified in combination with
--delete-dst-snapshots-except, then any destination snapshot retained by the rules above is actually only retained if it
also exists in the source dataset - all other destination snapshots are deleted. This is great for synchronization use
cases but should NEVER BE USED FOR LONG-TERM ARCHIVAL. Long-term archival use cases should instead specify the dummy
source dataset as they require an independent retention policy that is not tied to the current contents of the source
dataset.
--delete-dst-snapshots-except-plan DICT_STRING π
Retention periods to be used if pruning snapshots or bookmarks within the selected destination datasets via --delete-dst-snapshots. Has the same format as --create-src-snapshots-plan. Snapshots (--delete-dst-snapshots=snapshots) or bookmarks (with --delete-dst-snapshots=bookmarks) that do not match a period will be deleted. To avoid unexpected surprises, make sure to carefully specify ALL snapshot names and periods that shall be retained, in combination with --dryrun.
Example: "{'prod': {'onsite': {'secondly': 40, 'minutely': 40, 'hourly': 36, 'daily': 31, 'weekly': 12, 'monthly': 18, 'yearly': 5}, 'us-west': {'secondly': 0, 'minutely': 0, 'hourly': 36, 'daily': 31, 'weekly': 12, 'monthly': 18, 'yearly': 5}, 'eu-west': {'secondly': 0, 'minutely': 0, 'hourly': 36, 'daily': 31, 'weekly': 12, 'monthly': 18, 'yearly': 5}}, 'test': {'offsite': {'12hourly': 42, 'weekly': 12}, 'onsite': {'100millisecondly': 42}}}". This example will, for the
organization 'prod' and the intended logical target 'onsite', retain secondly snapshots that were created less than 40
seconds ago, yet retain the latest 40 secondly snapshots regardless of creation time. Analog for the latest 40 minutely
snapshots, latest 36 hourly snapshots, etc. It will also retain snapshots for the targets 'us-west' and 'eu-west' within
the 'prod' organization. In addition, within the 'test' organization, it will retain snapshots that are created every 12
hours and every week as specified, and name them as being intended for the 'offsite' replication target. Analog for
snapshots that are taken every 100 milliseconds within the 'test' organization. All other snapshots within the selected
datasets will be deleted - you've been warned!
The example scans the selected ZFS datasets for snapshots with names like prod_onsite_<timestamp>_secondly,
prod_onsite_<timestamp>_minutely, prod_us-west_<timestamp>_hourly, prod_us-west_<timestamp>_daily,
prod_eu-west_<timestamp>_hourly, prod_eu-west_<timestamp>_daily, test_offsite_<timestamp>_12hourly,
test_offsite_<timestamp>_weekly, and so on, and deletes all snapshots that do not match a retention rule.
Note: A zero within a period (e.g. 'hourly': 0) indicates that no snapshots shall be retained for the given period.
Note: --delete-dst-snapshots-except-plan is a convenience option that auto-generates a series of the following other options: --delete-dst-snapshots-except, --new-snapshot-filter-group, --include-snapshot-regex, --include-snapshot-times-and-ranks
--delete-empty-dst-datasets [{snapshots,snapshots+bookmarks}] π
Do nothing if the --delete-empty-dst-datasets option is missing or --recursive is missing. Otherwise, after successful replication step and successful --delete-dst-datasets and successful --delete-dst-snapshots steps, if any, delete any selected destination dataset that has no snapshot and no bookmark if all descendants of that destination dataset are also selected and do not have a snapshot or bookmark either (again, only if the existing destination dataset is selected via --{include|exclude}-dataset* policy). Never delete non-selected dataset subtrees or their ancestors.
For example, if the destination contains datasets h1,d1, and the include/exclude policy selects h1,d1, then check if h1,d1 can be deleted. On the other hand, if the include/exclude policy only selects h1 then only check if h1 can be deleted.
Note: Use --delete-empty-dst-datasets=snapshots to delete snapshot-less datasets even if they still contain bookmarks.
--monitor-snapshots DICT_STRING π
Do nothing if the --monitor-snapshots flag is missing. Otherwise, after all other steps, alert the user if the ZFS
'creation' time property of the latest snapshot for any specified snapshot name pattern within the selected datasets is too
old wrt. the specified age limit. The purpose is to check if snapshots are successfully taken on schedule, successfully
replicated on schedule, and successfully pruned on schedule. Process exit code is 0, 1, 2 on OK, WARNING, CRITICAL,
respectively. Example DICT_STRING: "{'prod': {'onsite': {'100millisecondly': {'latest': {'warning': '300 milliseconds', 'critical': '2 seconds'}}, 'secondly': {'latest': {'warning': '2 seconds', 'critical': '14 seconds'}}, 'minutely': {'latest': {'warning': '30 seconds', 'critical': '300 seconds'}}, 'hourly': {'latest': {'warning': '30 minutes', 'critical': '300 minutes'}}, 'daily': {'latest': {'warning': '4 hours', 'critical': '8 hours'}}, 'weekly': {'latest': {'warning': '2 days', 'critical': '8 days'}}, 'monthly': {'latest': {'warning': '2 days', 'critical': '8 days'}}, 'yearly': {'latest': {'warning': '5 days', 'critical': '14 days'}}, '10minutely': {'latest': {'warning': '0 minutes', 'critical': '0 minutes'}}}, '': {'daily': {'latest': {'warning': '4 hours', 'critical': '8 hours'}}}}}". This example alerts the user if
the latest src or dst snapshot named prod_onsite_<timestamp>_hourly is more than 30 minutes late (i.e. more than 30+60=90
minutes old) [warning] or more than 300 minutes late (i.e. more than 300+60=360 minutes old) [critical]. Analog for the
latest snapshot named prod_<timestamp>_daily, and so on.
Note: A duration that is missing or zero (e.g. '0 minutes') indicates that no snapshots shall be checked for the given snapshot name pattern.
Example output with --verbose:
--monitor_snapshots: OK. Latest snapshot for tank/foo@prod_<timestamp>_daily is 4.18h old: @prod_2025-01-10_08:30:05_daily
--monitor_snapshots: OK. Latest snapshot for tank/bar@prod_<timestamp>_daily is 4.18h old: @prod_2025-01-10_08:30:05_daily
--monitor_snapshots: Latest snapshot for tank/baz@prod_<timestamp>_daily is 1.2d old but should be at most 1.1d old: @prod_2025-01-09_08:30:05_daily
...
ERROR: Exiting bzfs with status code 2. Cause: --monitor_snapshots: Latest snapshot for tank/baz@prod_<timestamp>_daily is 1.2d old but should be at most 1.1d old: @prod_2025-01-09_08:30:05_daily
--monitor-snapshots-dont-warn π
--monitor-snapshots-dont-crit π
--compare-snapshot-lists [{src,dst,all,src+dst,src+all,dst+all,src+dst+all}] π
Do nothing if the --compare-snapshot-lists option is missing. Otherwise, after successful replication step and successful --delete-dst-datasets, --delete-dst-snapshots steps and --delete-empty-dst-datasets steps, if any, proceed as follows:
Compare source and destination dataset trees recursively wrt. snapshots, for example to check if all recently taken snapshots have been successfully replicated by a periodic job.
Example: List snapshots only contained in source (tagged with 'src'), only contained in destination (tagged with 'dst'),
and contained in both source and destination (tagged with 'all'), restricted to hourly and daily snapshots taken within the
last 7 days, excluding the last 4 hours (to allow for some slack/stragglers), excluding temporary datasets: bzfs tank1/foo/bar tank2/boo/bar --skip-replication --compare-snapshot-lists=src+dst+all --recursive --include-snapshot-regex '.*_(hourly|daily)' --include-snapshot-times-and-ranks '7 days ago..4 hours ago' --exclude-dataset-regex 'tmp.*'
This outputs a TSV file containing the following columns:
location creation_iso createtxg rel_name guid root_dataset rel_dataset name creation written
Example output row:
src 2024-11-06_08:30:05 17435050 /foo@test_2024-11-06_08:30:05_daily 2406491805272097867 tank1/src /foo tank1/src/foo@test_2024-10-06_08:30:04_daily 1730878205 24576
If the TSV output file contains zero lines starting with the prefix 'src' and zero lines starting with the prefix 'dst' then no source snapshots are missing on the destination, and no destination snapshots are missing on the source, indicating that the periodic replication and pruning jobs perform as expected. The TSV output is sorted by rel_dataset, and by ZFS creation time within each rel_dataset - the first and last line prefixed with 'all' contains the metadata of the oldest and latest common snapshot, respectively. Third party tools can use this info for post-processing, for example using custom scripts using 'csplit' or duckdb analytics queries.
The --compare-snapshot-lists option also directly logs various summary stats, such as the metadata of the latest common snapshot, latest snapshots and oldest snapshots, as well as the time diff between the latest common snapshot and latest snapshot only in src (and only in dst), as well as how many src snapshots and how many GB of data are missing on dst, etc.
Note: By default, if the source ZFS pool supports bookmarks, source bookmarks also participate in the comparison. A source bookmark and destination snapshot with the same GUID are considered to be contained in both locations, even if the corresponding source snapshot has already been deleted. Unmatched temporary bookmarks managed by bzfs remain hidden. Specify --no-use-bookmark to compare snapshots only.
Note: Consider omitting the 'all' flag to reduce noise and instead focus on missing snapshots only, like so: --compare-snapshot-lists=src+dst
Note: The source can also be an empty dataset, such as the hardcoded virtual dataset named 'dummy'.
Note: --compare-snapshot-lists is typically much faster than standard 'zfs list -t snapshot' CLI usage because the former issues requests with a higher degree of parallelism than the latter. The degree is configurable with the --threads option (see below).
--cache-snapshots π
If --cache-snapshots is specified, maintain a persistent local cache of recent snapshot creation times, recent successful replication times, and recent monitoring times, and compare them to a quick 'zfs list -t filesystem,volume -p -o snapshots_changed' to help determine if a new snapshot shall be created on the src, and if there are any changes that need to be replicated or monitored. Enabling the cache improves performance if --create-src-snapshots and/or replication and/or --monitor-snapshots is invoked frequently (e.g. every minute via cron) over a large number of datasets, with each dataset containing a large number of snapshots, yet it is seldom for a new src snapshot to actually be created, or there are seldom any changes to replicate or monitor (e.g. a snapshot is only created every day and/or deleted every day).
Note: This flag only has an effect on OpenZFS β₯ 2.2.
Note: This flag is only relevant for snapshot creation on the src if --create-src-snapshots-even-if-not-due is not specified.
--dryrun [{recv,send}], -n [{recv,send}] π
Do a dry run (aka 'no-op') to print what operations would happen if the command were to be executed for real (optional). This option treats both the ZFS source and destination as read-only. Accepts an optional argument for fine tuning that is handled as follows:
a) 'recv': Send snapshot data via 'zfs send' to the destination host and receive it there via 'zfs receive -n', which discards the received data there.
b) 'send': Do not execute 'zfs send' and do not execute 'zfs receive'. This is a less 'realistic' form of dry run, but much faster, especially for large snapshots and slow networks/disks, as no snapshot is actually transferred between source and destination. This is the default when specifying --dryrun.
Examples: --dryrun, --dryrun=send, --dryrun=recv
--verbose, -v π
-v -v -v flag, maybe along with --dryrun. All ZFS and
SSH commands (even with --dryrun) are logged such that they can be inspected, copy-and-pasted into a terminal shell and run
manually to help anticipate or diagnose issues. ERROR, WARN, INFO, DEBUG, TRACE output lines are identified by [E], [W],
[I], [D], [T] prefixes, respectively.--quiet, -q π
--no-privilege-elevation, -p π
Do not attempt to run state changing ZFS operations 'zfs create/rollback/destroy/send/receive/snapshot' as root (via 'sudo
-u root' elevation granted by administrators appending the following to /etc/sudoers: <NON_ROOT_USER_NAME> ALL=NOPASSWD:/path/to/zfs
Instead, the --no-privilege-elevation flag is for non-root users that have been granted corresponding ZFS permissions by administrators via 'zfs allow' delegation mechanism, like so: sudo zfs allow -u $SRC_NON_ROOT_USER_NAME snapshot,destroy,send,bookmark,hold $SRC_DATASET; sudo zfs allow -u $DST_NON_ROOT_USER_NAME mount,create,receive,rollback,destroy $DST_DATASET_OR_POOL.
If you do not plan to use the --force* flags and --delete-* CLI options then ZFS permissions 'rollback,destroy' can be
omitted, arriving at the absolutely minimal set of required destination permissions: mount,create,receive.
For extra security $SRC_NON_ROOT_USER_NAME should be different than $DST_NON_ROOT_USER_NAME, i.e. the sending Unix user on the source and the receiving Unix user at the destination should be separate Unix user accounts with separate private keys even if both accounts reside on the same machine, per the principle of least privilege.
Also see https://openzfs.github.io/openzfs-docs/man/master/8/zfs-allow.8.html#EXAMPLES and https://tinyurl.com/9h97kh8n and https://youtu.be/o_jr13Z9f1k?si=7shzmIQJpzNJV6cq
--no-stream π
--no-resume-recv π
Replication of snapshots via 'zfs send/receive' can be interrupted by intermittent network hiccups, reboots, hardware issues, etc. Interrupted 'zfs send/receive' operations are retried if the --retries and --retry-* options enable it (see above). In normal operation bzfs automatically retries such that only the portion of the snapshot is transmitted that has not yet been fully received on the destination. For example, this helps to progressively transfer a large individual snapshot over a wireless network in a timely manner despite frequent intermittent network hiccups. This optimization is called 'resume receive' and uses the 'zfs receive -s' and 'zfs send -t' feature.
The --no-resume-recv option disables this optimization such that a retry now retransmits the entire snapshot from scratch, which could slow down or even prohibit progress in case of frequent network hiccups. bzfs automatically falls back to using the --no-resume-recv option if it is auto-detected that the ZFS pool does not reliably support the 'resume receive' optimization.
Note: Snapshots that have already been fully transferred as part of the current 'zfs send/receive' operation need not be retransmitted regardless of the --no-resume-recv flag. For example, assume a single 'zfs send/receive' operation is transferring incremental snapshots 1 through 10 via 'zfs send -I', but the operation fails while transferring snapshot 10, then snapshots 1 through 9 need not be retransmitted regardless of the --no-resume-recv flag, as these snapshots have already been successfully received at the destination either way.
--create-bookmarks {all,hourly,minutely,secondly,none} π
For increased safety, bzfs replication behaves as follows wrt. ZFS bookmark creation, if it is autodetected that the source ZFS pool supports bookmarks:
all (default): Selects every source snapshot that will be sent during each 'zfs send' operation. This increases safety
at the expense of a little performance.
hourly: Selects each hourly, daily, weekly, monthly and yearly source snapshot that will be sent.
minutely and secondly: Same as hourly except that it also selects minutely and secondly snapshots, respectively.
none: No bookmark is created.
For every mode except none, the snapshot of a full send and the final snapshot of incremental replication are always
selected as they establish the initial and latest common bases, respectively.
For each selected source snapshot, bzfs creates a destination-specific temporary bookmark before starting its 'zfs send/receive' operation. After the operation succeeds, it renames the temporary bookmark to a finalized bookmark and deletes any obsolete temporary bookmarks for that destination. Both temporary and finalized bookmarks can serve as sources for future incremental replication once the corresponding snapshot exists on the destination. This guarantees continuity even if the bzfs process is killed after 'zfs send/receive' succeeds but before bookmark finalization.
Bookmarks exist so an incremental stream can continue to be sent from the source dataset without having to keep the already replicated snapshot around on the source dataset until the next upcoming snapshot has been successfully replicated. This way you can bookmark a snapshot on the source dataset, send the snapshot to another host, then delete the snapshot from the source dataset to save disk space, and then still incrementally send the next upcoming snapshot from the source dataset to the other host by referring to the bookmark.
The --create-bookmarks=none option disables this safety feature but is discouraged, because bookmarks are tiny and relatively cheap and help to ensure that ZFS replication can continue even if source and destination dataset somehow have no common snapshot anymore. For example, if a pruning script has accidentally deleted too many (or even all) snapshots on the source dataset in an effort to reclaim disk space, replication can still proceed because it can use the info in the bookmark (the bookmark must still exist in the source dataset) instead of the info in the metadata of the (now missing) source snapshot.
A ZFS bookmark is a tiny bit of metadata extracted from a ZFS snapshot by the 'zfs bookmark' CLI, and attached to a dataset, much like a ZFS snapshot. Note that a ZFS bookmark does not contain user data; instead a ZFS bookmark is essentially a tiny pointer in the form of the GUID of the snapshot and 64-bit transaction group number of the snapshot and creation time of the snapshot, which is sufficient to tell the destination ZFS pool how to find the destination snapshot corresponding to the source bookmark and (potentially already deleted) source snapshot. A bookmark can be fed into 'zfs send' as the source of an incremental send. Note that while a bookmark allows for its snapshot to be deleted on the source after successful replication, it still requires that its snapshot is not somehow deleted prematurely on the destination dataset, so be mindful of that. Also see https://www.youtube.com/watch?v=LaNgoAZeTww&t=316s.
By convention, a finalized bookmark created by bzfs has the same name as its corresponding snapshot, except the name also
contains the snapshot GUID for uniqueness. Destination-specific temporary bookmarks use names that begin with .TMPBZFS.;
this namespace is reserved for internal use. Do not create or delete such temporary bookmarks because bzfs automatically
manages and garbage collects them.
You can list bookmarks, like so: zfs list -t bookmark -o name,guid,createtxg,creation -d 1 $SRC_DATASET, and you can (and
should) periodically prune obsolete bookmarks just like snapshots, like so: zfs destroy $SRC_DATASET#$BOOKMARK.
Typically, bookmarks should be pruned less aggressively than snapshots, and destination snapshots should be pruned less
aggressively than source snapshots. As an example starting point, here is a command that deletes all non-temporary
bookmarks older than 90 days, but retains the latest 200 non-temporary bookmarks (per dataset) regardless of creation time:
bzfs dummy tank2/boo/bar --dryrun --recursive --skip-replication --delete-dst-snapshots=bookmarks --include-snapshot-times-and-ranks notime 'all except latest 200' --include-snapshot-times-and-ranks 'anytime..90 days ago'
--no-use-bookmark π
For increased safety, in normal replication operation bzfs replication also looks for bookmarks (in addition to snapshots) on the source dataset in order to find the most recent common snapshot wrt. the destination dataset, if it is auto-detected that the source ZFS pool supports bookmarks. The --no-use-bookmark option disables this safety feature but is discouraged, because bookmarks help to ensure that ZFS replication can continue even if source and destination dataset somehow have no common snapshot anymore.
Note that it does not matter whether a bookmark was created by bzfs or a third party script, as only the GUID of the
bookmark and the GUID of the snapshot is considered for comparison, and ZFS guarantees that any bookmark of a given
snapshot automatically has the same GUID, transaction group number and creation time as the snapshot. Apart from the
internal .TMPBZFS. namespace described above, you can create, delete and prune bookmarks any way you like, as bzfs
(without --no-use-bookmark) will happily work with whatever bookmarks currently exist, if any.
--ssh-cipher STRING π
^[email protected]--ssh-src-user STRING π
--ssh-dst-user STRING π
--ssh-src-host STRING π
--ssh-dst-host STRING π
--ssh-src-port INT π
--ssh-dst-port INT π
--ssh-src-config-file FILE π
--ssh-dst-config-file FILE π
--ssh-control-persist-secs INT π
--threads INT[%] π
--max-concurrent-ssh-sessions-per-tcp-connection INT π
For best throughput, bzfs uses multiple SSH TCP connections in parallel, as indicated by --threads (see above). For best startup latency, each such parallel TCP connection can carry a maximum of S concurrent SSH sessions, where S=--max-concurrent-ssh-sessions-per-tcp-connection (default: 8, min: 1). Concurrent SSH sessions are mostly used for metadata operations such as listing ZFS datasets and their snapshots. This client-side max sessions parameter must not be higher than the server-side sshd_config(5) MaxSessions parameter (which defaults to 10, see https://manpages.ubuntu.com/manpages/latest/man5/sshd_config.5.html).
Note: For better throughput, bzfs uses one dedicated TCP connection per ZFS send/receive operation such that the dedicated connection is never used by any other concurrent SSH session, effectively ignoring the value of the --max-concurrent-ssh-sessions-per-tcp-connection parameter in the ZFS send/receive case.
--r2r {off,pull,push} π
For remote-to-remote replication, controls whether the zfs send stream is relayed by the initiator or transferred
directly between source and destination in order to improve the throughput of bulk data transfers (default: off). This
option has no effect unless both replication endpoints are remote.
off: Keeps existing pull-push behavior where the initiator acts as an intermediary that relays the zfs send stream
between source and destination, which can become a central bandwidth bottleneck.Example: ssh alice@srchost 'zfs send ...' | ssh bob@dsthost 'zfs receive ...'
pull: Tells the destination host to pull the zfs send stream directly from the source host. Requires sh and ssh
on the destination host, plus SSH setup such that the destination user@host can ssh into the source host.Example: ssh bob@dsthost "sh -c 'ssh alice@srchost zfs send ... | zfs receive ...'"
push: Tells the source host to push the zfs send stream directly to the destination host. Requires sh and ssh on
the source host, plus SSH setup such that the source user@host can ssh into the destination host.Example: ssh alice@srchost "sh -c 'zfs send ... | ssh bob@dsthost zfs receive ...'"
Note: Orchestration still runs on the initiator; only the bulk data path changes. It is recommended to also set
--ssh-src-user and --ssh-dst-user explicitly in order to avoid potential confusion about which SSH user account
performs the nested remote-to-remote operation.
Note: If the required nested sh/ssh do not exist, bzfs falls back to --r2r=off. --r2r=pull falls back to
--r2r=off if --ssh-src-config-file is set to a non-empty value other than none, and --r2r=push falls back to
--r2r=off if --ssh-dst-config-file is set to a non-empty value other than none.
Note: --pv* progress reporting options have no effect in r2r modes; progress reporting is disabled.
--bwlimit STRING π
pv and mbuffer bandwidth rate limit for zfs send/receive data transfer (optional). Example: 100m to cap
throughput at 100 MB/sec. Default is unlimited. Also see https://manpages.ubuntu.com/manpages/latest/man1/pv.1.html--no-estimate-send-size π
--compression-program {zstd,lz4,pzstd,pigz,gzip,-} π
--compression-program-opts STRING π
--mbuffer-program {mbuffer,-} π
--mbuffer-program-opts STRING π
--ps-program {ps,-} π
--pv-program {pv,-} π
--pv-program-opts STRING π
--shell-program {sh,-} π
--ssh-program {ssh,hpnssh,-} π
--sudo-program {sudo,doas,-} π
--zpool-program {zpool,-} π
--log-dir DIR π
tail --follow=name --max-unchanged-stats=1 on both symlinks to follow what's
currently going on. Parallel replication generates a separate .pv file per thread. To monitor these, run something like
while true; do clear; for f in $(realpath $HOME/bzfs-logs/current/current.pv)*; do tac -s $(printf '\r') $f | tr '\r' '\n' | grep -m1 -v '^$'; done; sleep 1; done--log-file-prefix STRING π
${--log-dir}/${--log-file-prefix}<timestamp>${--log-file-infix}${--log-file-suffix}-<random>.log. Example:
--log-file-prefix=zrun_us-west_ --log-file-suffix=_daily will generate log file names such as
zrun_us-west_2024-09-03_12:26:15_daily-bl4i1fth.log--log-file-infix STRING π
${--log-dir}/${--log-file-prefix}<timestamp>${--log-file-infix}${--log-file-suffix}-<random>.log. Example:
--log-file-prefix=zrun_us-west_ --log-file-suffix=_daily will generate log file names such as
zrun_us-west_2024-09-03_12:26:15_daily-bl4i1fth.log--log-file-suffix STRING π
${--log-dir}/${--log-file-prefix}<timestamp>${--log-file-infix}${--log-file-suffix}-<random>.log. Example:
--log-file-prefix=zrun_us-west_ --log-file-suffix=_daily will generate log file names such as
zrun_us-west_2024-09-03_12:26:15_daily-bl4i1fth.log--log-subdir {daily,hourly,minutely} π
--log-syslog-address STRING π
--log-syslog-socktype {UDP,TCP} π
--log-syslog-facility INT π
--log-syslog-prefix STRING π
--log-syslog-level {CRITICAL,ERROR,WARN,INFO,DEBUG,TRACE} π
--include-envvar-regex REGEX [REGEX ...] π
On program startup, unset all Unix environment variables for which the full environment variable name matches at least one of the excludes but none of the includes. If an environment variable is included this decision is never reconsidered because include takes precedence over exclude. The purpose is to tighten security and help guard against accidental inheritance or malicious injection of environment variable values that may have unintended effects.
This option can be specified multiple times. A leading ! character indicates logical negation, i.e. the regex matches if
the regex with the leading ! character removed does not match. The default is to include no environment variables, i.e.
to make no exceptions to --exclude-envvar-regex. Example that retains at least these two env vars: --include-envvar-regex PATH --include-envvar-regex bzfs_min_pipe_transfer_size. Example that retains all environment variables without tightened
security: '.*'
--exclude-envvar-regex REGEX [REGEX ...] π
bzfs_.*--version π
Use these options to customize when snapshots that happen every N years are scheduled to be created on the source by the --create-src-snapshots option.
--yearly_year INT π
--yearly_month INT π
--yearly_monthday INT π
--yearly_hour INT π
--yearly_minute INT π
--yearly_second INT π
Use these options to customize when snapshots that happen every N months are scheduled to be created on the source by the --create-src-snapshots option.
--monthly_month INT π
--monthly_monthday INT π
--monthly_hour INT π
--monthly_minute INT π
--monthly_second INT π
Use these options to customize when snapshots that happen every N weeks are scheduled to be created on the source by the --create-src-snapshots option.
--weekly_weekday INT π
--weekly_hour INT π
--weekly_minute INT π
--weekly_second INT π
Use these options to customize when snapshots that happen every N days are scheduled to be created on the source by the --create-src-snapshots option.
--daily_hour INT π
--daily_minute INT π
--daily_second INT π
Use these options to customize when snapshots that happen every N hours are scheduled to be created on the source by the --create-src-snapshots option.
--hourly_minute INT π
--hourly_second INT π
Use these options to customize when snapshots that happen every N minutes are scheduled to be created on the source by the --create-src-snapshots option.
--minutely_second INT π
Use these options to customize when snapshots that happen every N seconds are scheduled to be created on the source by the --create-src-snapshots option.
--secondly_millisecond INT π
Use these options to customize when snapshots that happen every N milliseconds are scheduled to be created on the source by the --create-src-snapshots option.
--millisecondly_microsecond INT π
The following group of parameters specifies additional zfs receive '-o' options that can be used to configure copying of ZFS dataset properties from the source dataset to its corresponding destination dataset. The 'zfs-recv-o' group of parameters is applied before the 'zfs-recv-x' group.
--zfs-recv-o-targets {full,incremental,full+incremental} π
--zfs-recv-o-sources STRING π
--zfs-recv-o-include-regex [REGEX ...] π
Take the output properties of --zfs-recv-o-sources (see above) and filter them such that we only retain the properties whose name matches at least one of the --include regexes but none of the --exclude regexes. If a property is excluded this decision is never reconsidered because exclude takes precedence over include. Append each retained property to the list of '-o' options in --zfs-recv-program-opt(s), unless another '-o' or '-x' option with the same name already exists therein. In other words, --zfs-recv-program-opt(s) takes precedence.
Zero or more regexes can be specified. Specify zero regexes to append no extra '-o' option. A leading ! character
indicates logical negation, i.e. the regex matches if the regex with the leading ! character removed does not match. If
the option starts with a + prefix then regexes are read from the newline-separated UTF-8 text file given after the +
prefix, one regex per line inside of the text file. The basename must contain the substring 'bzfs_argument_file'.
The default regex is
'aclinherit|aclmode|acltype|atime|checksum|compression|copies|logbias|primarycache|recordsize|redundant_metadata|relatime|secondarycache|snapdir|sync|xattr'.
Example: --zfs-recv-o-include-regex compression recordsize. More examples: .* (include all properties), foo bar myapp:.* (include three regexes) +zfs-recv-o_regexes_bzfs_argument_file.txt,
+/path/to/zfs-recv-o_regexes_bzfs_argument_file.txt
See https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html
--zfs-recv-o-exclude-regex REGEX [REGEX ...] π
The following group of parameters specifies additional zfs receive '-x' options that can be used to configure copying of ZFS dataset properties from the source dataset to its corresponding destination dataset. The 'zfs-recv-o' group of parameters is applied before the 'zfs-recv-x' group.
--zfs-recv-x-targets {full,incremental,full+incremental} π
--zfs-recv-x-sources STRING π
--zfs-recv-x-include-regex [REGEX ...] π
Take the output properties of --zfs-recv-x-sources (see above) and filter them such that we only retain the properties whose name matches at least one of the --include regexes but none of the --exclude regexes. If a property is excluded this decision is never reconsidered because exclude takes precedence over include. Append each retained property to the list of '-x' options in --zfs-recv-program-opt(s), unless another '-o' or '-x' option with the same name already exists therein. In other words, --zfs-recv-program-opt(s) takes precedence.
Zero or more regexes can be specified. Specify zero regexes to append no extra '-x' option. A leading ! character
indicates logical negation, i.e. the regex matches if the regex with the leading ! character removed does not match. If
the option starts with a + prefix then regexes are read from the newline-separated UTF-8 text file given after the +
prefix, one regex per line inside of the text file. The basename must contain the substring 'bzfs_argument_file'.
The default is to include no properties, thus by default no extra '-x' option is appended. Example:
--zfs-recv-x-include-regex compression recordsize. More examples: .* (include all properties), foo bar myapp:.*
(include three regexes) +zfs-recv-x_regexes_bzfs_argument_file.txt, +/path/to/zfs-recv-x_regexes_bzfs_argument_file.txt
See https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html
--zfs-recv-x-exclude-regex REGEX [REGEX ...] π
2,247 followers Β· starred Apr 2026
185 followers Β· starred Jan 2026
2,273 followers Β· starred Sep 2026
125 followers Β· starred Jan 2026