hf-mount

Mount Hugging Face Buckets and repos as local filesystems. No download, no copy, no waiting.

hf-mount start bucket myuser/my-bucket /tmp/data

Also works with any model or dataset repo (read-only):

hf-mount start repo openai/gpt-oss-20b /tmp/gpt-oss

Commands will pick up your HF_TOKEN from the environment, or you can pass it explicitly with --hf-token.

Then use your local folders as usual:

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("/tmp/gpt-oss")  # reads on demand, no download step

hf-mount exposes Hugging Face Buckets and Hub repos as a local filesystem via FUSE or NFS. Files are fetched lazily on first read, so only the bytes your code actually touches ever hit the network.

Two backends are available:

• NFS (recommended) -- works everywhere, no root, no kernel extension
• FUSE -- tighter kernel integration, requires root or macFUSE on macOS

Agentic storage: Agents don't require complex APIs or SDKs, they thrive on the filesystem: ls, cat, find, grep, and the power of composable UNIX pipelines.

Install

Homebrew (macOS, Linux)

brew install hf-mount

On macOS, this installs the NFS backend only (hf-mount, hf-mount-nfs). For the FUSE backend on macOS, download the binary manually or build from source — macFUSE is closed-source and not distributable through homebrew-core.

Manual download

Binaries are available on GitHub Releases:

System dependencies (FUSE only)

The NFS backend has no system dependencies. For FUSE:

Linux: sudo apt-get install -y fuse3 (pre-built binaries only need the runtime; building from source also requires libfuse3-dev)

macOS: install macFUSE (brew install macfuse, requires reboot on first install)

Build from source

Requires Rust 1.89+.

# NFS only (no system deps, works everywhere)
cargo build --release --features nfs

# FUSE (requires macFUSE on macOS, fuse3 on Linux)
cargo build --release --features fuse

# All backends
cargo build --release --features fuse,nfs

Binaries: target/release/hf-mount, target/release/hf-mount-nfs, target/release/hf-mount-fuse

Best for / Not for

Best for:

• Loading models and datasets without downloading the full repo
• Browsing repo contents (ls, cat, find) without cloning
• Read-heavy ML workloads (training, inference, evaluation)
• Environments where disk space is limited

Not for:

• General-purpose networked filesystem (no multi-writer support, no cross-node file locking)
• Latency-sensitive random I/O (first reads require network round-trips)
• Workloads that need strong consistency (files can be stale for up to 10 s)
• Heavy concurrent writes from multiple mounts (last writer wins, no conflict detection)
• Editing files with text editors in default (streaming) mode (use --advanced-writes)

Advisory file locks (flock, fcntl POSIX record locks) are supported locally on a single mount on both backends — enough for Python filelock, huggingface_hub, datasets, and similar cache-coordination use cases within one machine. They are not coordinated across multiple clients.

See Consistency model for details.

Usage

Mount a repo (read-only)

# Public model (no token needed)
hf-mount start repo openai/gpt-oss-20b /tmp/model

# Private model
hf-mount start --hf-token $HF_TOKEN repo myorg/my-private-model /tmp/model

# Dataset
hf-mount start repo datasets/open-index/hacker-news /tmp/hn

# Specific revision
hf-mount start repo openai-community/gpt2 /tmp/gpt2 --revision v1.0

# Subfolder only
hf-mount start repo openai-community/gpt2/onnx /tmp/onnx

Mount a Bucket (read-write)

Buckets are S3-like object storage on the Hub, designed for large-scale mutable data (training checkpoints, logs, artifacts) without git version control.

hf-mount start --hf-token $HF_TOKEN bucket myuser/my-bucket /tmp/data

# Read-only
hf-mount start --hf-token $HF_TOKEN --read-only bucket myuser/my-bucket /tmp/data

# Subfolder only
hf-mount start --hf-token $HF_TOKEN bucket myuser/my-bucket/checkpoints /tmp/ckpts

Manage mounts

hf-mount status                  # list running mounts
hf-mount stop /tmp/data          # stop and unmount

Logs are written to ~/.hf-mount/logs/. PID files are stored in ~/.hf-mount/pids/.

FUSE backend

By default, hf-mount uses NFS. Pass --fuse for tighter kernel integration (page cache invalidation, per-file metadata revalidation). Requires fuse3 on Linux or macFUSE on macOS.

hf-mount start --fuse --hf-token $HF_TOKEN bucket myuser/my-bucket /mnt/data

Foreground mode

For scripts, containers, or debugging, use the backend binaries directly (they run in the foreground):

hf-mount-nfs repo gpt2 /tmp/gpt2
hf-mount-fuse --hf-token $HF_TOKEN bucket myuser/my-bucket /mnt/data

macOS: launch as a daemon with launchd

To have hf-mount start automatically on login, create a LaunchAgent:

label=co.huggingface.hf-mount

mkdir -p ~/Library/LaunchAgents

cat > ~/Library/LaunchAgents/$label.plist <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>$label</string>
    <key>ProgramArguments</key>
    <array>
        <string>$HOME/.local/bin/hf-mount-nfs</string>
        <string>repo</string>
        <string>openai/gpt-oss-20b</string>
        <string>/tmp/gpt-oss</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>StandardOutPath</key>
    <string>/tmp/hf-mount.log</string>
    <key>StandardErrorPath</key>
    <string>/tmp/hf-mount.log</string>
</dict>
</plist>
EOF

launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/$label.plist

To stop: launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/$label.plist

Unmount

umount /tmp/data                 # NFS or FUSE (macOS)
fusermount -u /tmp/data          # FUSE (Linux)
hf-mount stop /tmp/data          # daemon mounts

Graceful shutdown (SIGTERM / CSI sidecar)

On SIGTERM the sidecar bounds the dirty-data flush (--flush-shutdown-timeout-ms) and disarms the kernel-cache invalidators before draining. This is deliberate: a FUSE_NOTIFY_INVAL_INODE writev issued during teardown can block uninterruptibly in the kernel (waiting on a folio under writeback the exiting daemon can no longer complete), leaving a D-state thread that even exit_group can't reap — an unkillable pod. Disarming the invalidator avoids creating that wedge; the CSI driver aborting the FUSE connection on NodeUnpublishVolume is the kernel-level backstop for any already-in-flight notify.

The same FUSE_NOTIFY_INVAL_INODE writev can also wedge at runtime (not just on shutdown): when the poll loop detects a remote change to a file the app currently has open, a full page-cache invalidation blocks in-kernel on a folio lock held by the app's in-flight read(), which is itself waiting for the daemon — a deadlock. Two guards prevent this: invalidations targeting an inode with open handles drop attributes only (a negative-offset notify the kernel never lets touch pages), and the blocking writev runs on the runtime's blocking pool rather than a core worker, so it can never starve FUSE request servicing. The trade-off: a file with a long-lived open handle won't see remote content updates refreshed in its page cache until the handle closes. On close-and-reopen the kernel revalidates the (attr-only-invalidated) attributes and, via the negotiated FUSE_AUTO_INVAL_DATA, drops the stale pages itself when it sees the new mtime/size — so a stale read only persists if a remote content change preserves both mtime and size on a file that was open at the moment of the change.

Options

Bounding inode memory

Under workloads that enumerate large trees (a find, a documentation scraper, du -sh), the in-memory inode table can grow without bound: every path the kernel ever looked up stays resident. With --inode-soft-limit N set, two evictors cooperate to keep the table near N:

1. Insert-time evictor (synchronous): before adding a new entry when len() >= N + 256, drop the oldest-touched file/symlink/leaf-directory entries.
   • Polite mode: only entries the kernel has already released (forget-ed). Safe, no FUSE races.
   • Force mode: when above 2 × N and polite found nothing, drop entries even if the kernel still caches the dentry. A racing kernel op sees ENOENT and re-looks up. Dirty files, locally-created dirs/symlinks, and inodes with live file handles are never dropped — the force path preserves all user data.
2. Background LRU sweep (every --lru-sweep-interval-ms): for inodes the kernel has cached but our table doesn't want, send FUSE_NOTIFY_INVAL_ENTRY so the kernel drops its dentry and sends us forget. Bounded to 1024 invalidations per sweep with EAGAIN backoff so we don't flood the notify channel.

Tuning: pick N below what a full-tree enumeration of your bucket would produce. For hf-doc-build/doc-dev with ~20k files, --inode-soft-limit 10000 keeps sidecar RSS ~250 MiB with a 1 GiB cgroup cap.

Overlay mode

--overlay makes the mount point itself a writable local layer on top of the remote source. Reads return whatever the remote has, plus anything you've put on local disk under the mount point. Writes go only to local disk — the remote is never touched. Local files survive an unmount/remount.

Useful when several machines or processes need to share a read-only remote view but each layer their own files on top — for example, a shared compilation cache where producer machines populate a bucket with compiled artifacts (torch.compile, vLLM, JAX/XLA, AWS Neuron) and every consumer mounts the same bucket with --overlay. Cache hits are served from the bucket without recompiling; cache misses compile locally and stay on the local disk, never pushed back to the bucket.

# Producer (writes compiled artifacts to the bucket — regular bucket mount)
hf-mount start bucket myorg/torch-compile-cache "$TORCHINDUCTOR_CACHE_DIR"

# Consumer (reads from the bucket, compiles locally on miss)
hf-mount start --overlay bucket myorg/torch-compile-cache "$TORCHINDUCTOR_CACHE_DIR"

What you can do:

• Read every file from the remote source.
• Read every file already present in the local layer; when a name exists in both, the local copy wins.
• Create new files and directories — they land in the local layer.
• Modify, rename, delete, or chmod any file or directory that lives in the local layer.

What you can't do:

• Modify, rename, delete, or chmod a file that exists only on the remote. These operations fail with a permission error. To diverge from a remote file, copy it under a new name through the mount; the copy is a regular local file you own.
• Shadow an existing remote name with a new local file once the mount is active. If you need a local file at a name that already exists on the remote, drop it in the mount-point directory before starting the mount — pre-existing files at the mount point stay visible and take precedence.
• Place symlinks in the local layer and expect them to show up. Symlinks are hidden from the merged view so the mount can't be tricked into reading or writing outside the mount point.

Logging

RUST_LOG=hf_mount=debug hf-mount-fuse repo gpt2 /mnt/gpt2

Features

• FUSE & NFS backends -- FUSE for standard Linux/macOS, NFS for environments without /dev/fuse
• Lazy loading -- files are fetched on demand, not eagerly downloaded
• Subfolder mounting -- mount only a subdirectory (e.g. user/model/ckpt/v2)
• Simple writes (default) -- append-only, in-memory, synchronous upload on close
• Advanced writes (--advanced-writes) -- staging files on disk, random writes + seek, async debounced flush
• Remote sync -- background polling detects remote changes and updates the local view
• POSIX metadata -- chmod, chown, timestamps, symlinks (in-memory only, lost on unmount)
• Overlay mode (--overlay) -- mount point doubles as a writable local layer; remote stays read-only

Consistency model

hf-mount provides eventual consistency with remote changes. There is no push notification from the Hub; all freshness relies on client-side polling.

Reads

Files can be stale for up to --metadata-ttl-ms (default 10 s) after a remote update. Two mechanisms detect changes:

1. Metadata revalidation (FUSE only) -- when the per-file TTL expires, the next access checks the Hub. If the file changed, cached data is invalidated.
2. Background polling (default every 30 s) -- lists the full tree and detects additions, modifications, and deletions.

Writes

Streaming mode buffers writes in memory and uploads on close(). A crash before close means data loss.

Note: Streaming mode does not support text editors (vim, nano, emacs). Editors that use unlink+create save patterns will be blocked (EPERM) to prevent data loss. Use --advanced-writes for interactive editing.

Advanced mode downloads the full file to local disk before allowing edits. After close(), dirty files are flushed asynchronously. A crash before flush completes means data loss.

FUSE vs NFS

How it works

hf-mount sits between your application and the Hugging Face Hub. It presents a standard filesystem interface (FUSE or NFS) and translates file operations into Hub API calls and storage fetches.

Reads go through an adaptive prefetch buffer that starts small and grows with sequential access. Writes are uploaded to HF storage and committed via the Hub API. A background poll loop keeps the local view in sync with remote changes.

Built on xet-core for content-addressed storage and efficient file transfers, and fuser for the FUSE implementation.

Kubernetes

Use the hf-csi-driver to mount Buckets and repos as Kubernetes volumes. The CSI driver runs hf-mount inside a DaemonSet and exposes mounts to pods via the Container Storage Interface.

helm install hf-csi oci://ghcr.io/huggingface/charts/hf-csi-driver

See the hf-csi-driver README for setup and examples.

Testing

# Unit tests (no network, no token)
cargo test --lib --features fuse,nfs

# Integration tests (require HF_TOKEN and FUSE)
HF_TOKEN=... cargo test --release --features fuse,nfs --test fuse_ops -- --test-threads=1 --nocapture
HF_TOKEN=... cargo test --release --features fuse,nfs --test nfs_ops -- --test-threads=1 --nocapture

# Repo mount test (public repo, no token needed)
cargo test --release --features nfs --test repo_ops -- --test-threads=1 --nocapture

# Benchmarks
HF_TOKEN=... cargo test --release --features fuse,nfs --test bench -- --nocapture

Troubleshooting

I'm getting Operation not permitted on MacOS while opening or listing files in a mounted bucket using VSCode

You need to enable "Full disk access" to your VSCode (System settings > Privacy > Full disk access).

License

Apache-2.0