DAXFS

Disaggregated filesystem for multi-kernel and multi-host shared memory.

DAXFS operates directly on DAX-capable memory (persistent memory, CXL memory, or DMA buffers) via direct load/store access. Multiple independent kernels or CXL-connected hosts sharing memory get a unified storage layer: shared namespace, cooperative page cache, and zero-copy CPU/GPU access over aggregated distributed storage.

Not for traditional disks. DAXFS requires byte-addressable memory with DAX support. It cannot run on block devices; the entire design assumes direct memory pointer access and synchronization with cmpxchg.

Features

• Zero-copy reads - Direct memory access, no page cache overhead
• Lock-free writes - CAS-based hash overlay, no locks between participants in one cache-coherence domain (see Coherence model)
• Shared page cache - Demand-paged cache in DAX memory, visible to all participants sharing a cache-coherent domain
• Multi-kernel namespace - Each kernel instance exports local storage into a shared filesystem (multikernel, single coherence domain); cross-host CXL is gated future work (see Coherence model)
• Flexible backing - Physical address, DAX device, or dma-buf
• Security by simplicity - Flat directory format, bounded validation, no pointer chasing

Use Cases

• LLM inference serving - Multiple GPU kernels share model weights through daxfs; one copy in shared memory serves all instances, cold start goes from minutes to seconds
• Multikernel/multi-host - Shared rootfs across kernel instances or CXL-connected hosts with cooperative caching
• CXL memory pooling - Common filesystem across CXL-connected hosts with lock-free concurrent access
• GPU/accelerator - Zero-copy access to data via dma-buf
• Container rootfs - Shared base image with writable overlay per container

Why Not ...

DAXFS vs FamFS

Both DAXFS and FamFS target CXL shared memory, but they differ fundamentally in architecture:

FamFS is a thin mapping layer that exposes pre-allocated files on shared memory. DAXFS is a general-purpose shared in-memory filesystem that uses shared-memory atomics for lock-free coordination within a cache-coherence domain: concurrent writes, cooperative caching, and layered storage without a central coordinator. Cross-host CXL operation requires hardware coherence and is gated future work; see Coherence model.

Building

make              # build kernel module + tools
make clean

Requires Linux 5.11+ and CONFIG_FS_DAX enabled in the target kernel.

Usage

# Create a static read-only image
mkdaxfs -d /path/to/rootfs -o image.daxfs

# Create and mount from DMA heap (read-only)
mkdaxfs -d /path/to/rootfs -H /dev/dma_heap/system -s 256M -m /mnt

# Split mode: metadata+overlay+cache in DAX, file data in backing file (writable)
mkdaxfs -d /path/to/rootfs -H /dev/dma_heap/mk -m /mnt -o /data/rootfs.img

# Empty mode: writable filesystem with no base image
mkdaxfs --empty -H /dev/dma_heap/mk -m /mnt -s 256M

# Custom overlay sizing
mkdaxfs -d /path/to/rootfs -o image.daxfs -O 128M -B 131072

# Create at physical address, then mount separately
mkdaxfs -d /path/to/rootfs -p 0x100000000 -s 256M
mount -t daxfs -o phys=0x100000000,size=0x10000000 none /mnt

# Split mode mount with backing file
mount -t daxfs -o phys=ADDR,size=SIZE,backing=/data/rootfs.img none /mnt

mkdaxfs Options

Mount Options

phys=ADDR, size=SIZE, validate (check untrusted data), backing=PATH (backing file for split mode).

For dma-buf backing, use the new mount API (fsopen/fsconfig/fsmount) with FSCONFIG_SET_FD to pass the dma-buf fd.

Inspection

# Show memory layout and status
daxfs-inspect status -m /mnt/daxfs

# Show overlay hash table details (bucket utilization, entry types, pool usage)
daxfs-inspect overlay -m /mnt/daxfs

# Inspect via physical address
daxfs-inspect status -p 0x100000000 -s 256M

Architecture

Modes

Hash Overlay

The overlay replaces traditional journaling or log-structured writes with a CAS-based hash table on DAX memory. Multiple kernels or CXL hosts can write concurrently with no locks.

• Open addressing with linear probing, 16-byte buckets
• Atomic insert via cmpxchg on bucket's state_key field (FREE→USED)
• Bump allocator for pool entries (atomic fetch-and-add on pool_alloc)
• Entry types: inode metadata, data pages (4KB COW), directory entries with tombstone deletion

Key encoding (63 bits):

• Data: (ino << 20) | pgoff (up to 1M pages per file)
• Inode: (ino << 20) | 0xFFFFF (sentinel pgoff)
• Dirent: FNV-1a(parent_ino, name) (63-bit hash)

Read path: overlay → base image → pcache (backing store). Write path: COW from base image into overlay data page.

Shared Page Cache

Direct-mapped cache in DAX memory for backing store mode. Within a single hardware cache-coherence domain (e.g. multiple kernel instances on one coherent machine), the cache is visible to all participants via hardware coherence with no software coherency protocol. Cross-host CXL sharing requires hardware coherence (CXL 3.0) and is gated future work; see Coherence model.

• 3-state machine: FREE → PENDING → VALID, all transitions via cmpxchg
• Multi-file tags: tag = (ino << 20) | pgoff, multiple backing files share one cache
• Host fills, spawns wait: host kernel reads backing file into PENDING slots; spawn kernels busy-poll until VALID
• Pre-warming: mkdaxfs pre-populates cache slots at image creation time

On-Disk Format

Defined in include/daxfs_format.h (version 7).

Base image (flat format):

• Inode table: fixed 64-byte entries
• Data area: file contents + directory entry arrays
• Directories store daxfs_dirent arrays (271 bytes each, 255-char max name)

Overlay (hash table):

• Header (4KB): magic, bucket count, pool offsets, atomic counters
• Bucket array: bucket_count × 16 bytes, open addressing
• Pool: variable-size entries (inodes 32B, data pages 4104B, dirents ~280B)

Page cache:

• Header (4KB): magic, slot count, offsets, pending counter
• Slot metadata: slot_count × 16 bytes (state_tag + ref_bit)
• Slot data: slot_count × 4KB pages

Security

DAXFS uses a flat directory format designed for safe handling of untrusted images:

Limitations

• No mknod support (device nodes, FIFOs, sockets not supported)
• Filename max 255 characters (matches VFS NAME_MAX)
• Overlay pool entries are recycled via per-type free lists, but the pool itself is not compacted
• Multi-file pcache tag supports up to ~1M pages per file (4GB with 4KB pages)
• Overlay hash table size is fixed at creation time