ADIOS2 : The Adaptable Input Output System version 2

ADIOS2 is an adaptable, scalable, and unified framework for scientific data I/O. It transports self-describing variables and attributes across files, networks, and memory using a single API, and is used in production on the world's largest supercomputers.

Features

• Unified I/O API: same interface for files, wide-area networks, and in-memory staging; switch transport without changing application code
• BP format: self-describing binary-packed format (.bp) with rapid metadata extraction and built-in compression support
• MPI-native: scales from laptop to petabyte workloads on the largest HPC systems; serial use also supported
• Streaming and step-based: asynchronous, grouped variable transport with an explicit steps abstraction for in-situ and in-transit workflows
• Multiple language bindings: C++17, Python, C99, Fortran 90, and Matlab
• Compression operators: ZFP, SZ3, MGARD, Blosc2, PNG, and others
• Encryption operators: symmetric (AES-256-GCM) and asymmetric encryption available for data at rest
• In-situ/in-transit engines: SST and DataMan engines for code coupling over RDMA, MPI, and TCP without touching files
• DAOS engine: writes and reads datasets through the DAOS data plane (native object API) directly, with no dfuse or POSIX interception required
• Plugin support: extend ADIOS2 with custom engines and operators via shared-library plugins at runtime
• Security-focused: committed to OpenSSF best practices; see the OpenSSF Scorecard and Best Practices badge
• Widely available: packaged for major Linux distributions, Conda, Spack, Homebrew, and vcpkg; validated on U.S. DoE supercomputers including Frontier, Summit, and Perlmutter
• HDF5 VOL: existing HDF5 applications can use ADIOS2 as a backend transparently via the HDF5 Virtual Object Layer

Example

import adios2
import numpy as np

data = np.array([1.0, 2.0, 3.0, 4.0, 5.0])

# Write to a BP file
with adios2.Stream("data.bp", "w") as f:
    f.write("data", data)

# Read it back
with adios2.Stream("data.bp", "r") as f:
    for _ in f.steps():
        result = f.read("data")

print(result)  # [1. 2. 3. 4. 5.]

Install ADIOS2

Python (PyPI):

pip install adios2

From source: See the install documentation and scripts/runconf/runconf.sh for a CMake configuration example. Once installed, refer to Linking ADIOS2.

Packages

An Apptainer/Singularity SIF image is also published to GHCR on every master push:

apptainer pull adios2.sif oras://ghcr.io/ornladios/adios2-apptainer-sif-master-x64:latest

Notes for packagers other than HPC sysadmins

ADIOS2 has a wide spectrum of optional features, from a small, dependency-free core to specialized backends that pull in heavy or uncommon third-party libraries. It is also a research project, so those backends come and go, and the sites that rely on them usually build from source for each environment (or via Spack) rather than from a package manager. Therefore we recommend a lean build, not trying to satisfy all possible optional dependencies.

Recommended tiers

Core (no or trivial dependencies):

• BP3 / BP4 / BP5 file engines (.bp format), no external dependencies
• SST streaming engine (TCP by default)
• C and C++ bindings

Common options (widely packaged):

• MPI, recommended but not required
• Python, Fortran bindings
• HDF5 (interop and the HDF5 VOL)
• Compression: ZFP, SZ3, BZip2, Blosc2, PNG
• Campaign (requires SQLite3 and ZLIB, both lightweight and commonly available)
• CURL (HTTPS remote data access)

Specialized (of interest only to specific workflows):

• Additional compression / refactoring: MGARD, SZ (for existing data only), LibPressio, BigWhoop, ProDM
• GPU: CUDA, Kokkos
• Encryption: Sodium (libsodium; backs both the EncryptionOperator and encrypted Campaign metadata)
• Remote / storage backends: XRootD, OpenSSL (TLS for HTTPS remote access), AWSSDK (S3), DAOS
• In-situ visualization: Catalyst (ParaView)

Legacy engines and transports (unmaintained, off by default):

• DataMan, SSC, MHS, DataSpaces, IME

These older staging and coupling engines, and the DDN IME transport, are no longer actively maintained and are candidates for future removal. They are disabled by default and must be enabled explicitly (for example -DADIOS2_USE_MHS=ON) if needed. ZeroMQ is required only by the DataMan engine and is likewise off by default, so a default build pulls in no ZeroMQ dependency.

HPC network transports (RDMA)

The SST engine reaches full performance through RDMA data planes that depend on fabric libraries: libfabric (InfiniBand/verbs and HPE-Cray Slingshot/CXI; detected automatically when SST is enabled), UCX (ADIOS2_USE_UCX), and Mercury (ADIOS2_USE_Mercury). These matter on systems with an RDMA interconnect, where an HPC-targeted package should install one or more of libfabric, UCX and Mercury before building so SST picks them up. On machines without an RDMA network they provide no benefit may cause trouble (SST will fallback to TCP and or MPI dataplanes if RDMA is not directly available.) General-purpose packages can safely leave them out.

A default build

A default configuration is already lean: the legacy engines above are off, and every other optional feature is enabled only when its dependency is already present. Enabling the common options is a matter of turning them on explicitly, for example:

cmake -DADIOS2_USE_HDF5=ON -DADIOS2_USE_Python=ON -DADIOS2_USE_MPI=ON ..

Questions about whether a given feature is worth packaging are welcome on GitHub Discussions.

Resources

• Documentation: adios2.readthedocs.io
• Latest release: v2.12.1
• All releases: github.com/ornladios/ADIOS2/releases
• Citing: if you find ADIOS2 useful, please cite our SoftwareX paper

Community

ADIOS2 is an open source project: questions, discussion, and contributions are welcome.

• Mailing list: adios-ecp@kitware.com
• Github Discussions: https://github.com/ornladios/ADIOS2/discussions
• Bug reports: open an issue
• Contributing: see the Contributor's Guide

Institutions

ADIOS2 is developed as a multi-institutional collaboration between:

• Oak Ridge National Laboratory
• Kitware Inc.
• Lawrence Berkeley National Laboratory
• Georgia Institute of Technology
• Rutgers University

License

ADIOS2 is licensed under the Apache License v2.0. See the accompanying Copyright.txt for more details.