📥 ElBruno.HuggingFace.Downloader

A .NET library and CLI tool to download files (ONNX models, tokenizers, voice presets, etc.) from Hugging Face Hub repositories with progress reporting, caching, and authentication support.

Features

• 📦 Download any file from public or private Hugging Face repositories
• 📊 Rich progress reporting with stages (Checking → Downloading → Validating → Complete)
• 🔑 HF_TOKEN authentication for gated/private repositories (env var or explicit)
• 🌐 Custom Hub endpoints via HF_ENDPOINT or downloader options
• 🔒 Atomic writes using temp files to avoid partial/corrupt downloads
• 🔁 Resumable downloads using HTTP range requests for interrupted large files
• 📌 Revision pinning with resolved commit metadata and optional expected commit enforcement
• 🗂️ Remote file listing — enumerate files in a repo (models, datasets, spaces) before downloading, no need to know exact paths in advance
• ✅ Required vs optional files — optional files fail silently
• 🧾 Manifest-based bundles with SHA-256 and size validation
• 📏 HEAD requests to resolve total download size before starting
• ⏭️ Skip existing files — only downloads what's missing
• 🖥️ Cross-platform cache directory helpers (Windows, Linux, macOS)
• 💉 DI-friendly with IServiceCollection extension methods
• 📝 ILogger integration for structured logging

📦 NuGet Packages

Installation

Library (NuGet)

dotnet add package ElBruno.HuggingFace.Downloader

CLI Tool

dotnet tool install -g ElBruno.HuggingFace.Downloader.Cli

Once installed, use the hfdownload command:

# Download model files
hfdownload download sentence-transformers/all-MiniLM-L6-v2 onnx/model.onnx tokenizer.json

# Download a specific tag into a revision-aware cache directory
hfdownload download sentence-transformers/all-MiniLM-L6-v2 onnx/model.onnx --revision v1.0

# Restart from scratch instead of resuming a preserved partial download
hfdownload download sentence-transformers/all-MiniLM-L6-v2 onnx/model.onnx --no-resume

# Ensure a manifest-defined bundle with integrity validation
hfdownload download --manifest ./phi4-bundle.json -o ./models/phi4

# Check if files exist locally
hfdownload check sentence-transformers/all-MiniLM-L6-v2 onnx/model.onnx tokenizer.json

# List remote files in a repo without knowing filenames up front
hfdownload files sentence-transformers/all-MiniLM-L6-v2

# List remote files in a dataset repo, as JSON
hfdownload files my-org/my-dataset --repo-type dataset --format json

# List cached models
hfdownload list

# Delete a cached model
hfdownload delete sentence-transformers/all-MiniLM-L6-v2

# See all commands
hfdownload --help

See the full CLI Reference for all commands and options.

Quick Start (Library)

1) Download model files

using ElBruno.HuggingFace;

using var downloader = new HuggingFaceDownloader();

var request = new DownloadRequest
{
    RepoId = "sentence-transformers/all-MiniLM-L6-v2",
    LocalDirectory = "./models/miniLM",
    RequiredFiles = ["onnx/model.onnx", "tokenizer.json"],
    OptionalFiles = ["tokenizer_config.json", "vocab.txt"],
    Revision = "refs/pr/42"
};

await downloader.DownloadFilesAsync(request);

Console.WriteLine($"Resolved commit: {request.ResolvedCommitSha}");

2) Ensure a manifest-based bundle with SHA-256 validation

var manifest = new ModelBundleManifest
{
    RepoId = "microsoft/Phi-4-mini-instruct-onnx",
    Revision = "main",
    Files =
    [
        new ModelBundleFile
        {
            Path = "onnx/model.onnx",
            Size = 123456789,
            Sha256 = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
        },
        new ModelBundleFile
        {
            Path = "tokenizer.json",
            Required = false
        }
    ]
};

var bundle = await downloader.EnsureBundleAsync(manifest, "./models/phi4");
Console.WriteLine(bundle.ResolvedManifestPath);

Manifest-driven bundle downloads validate existing files before reuse, re-download corrupted files, and write a resolved manifest to ./models/phi4/.hf.bundle.resolved.json.

3) Pin a branch or tag to the expected commit

var request = new DownloadRequest
{
    RepoId = "my-org/my-model",
    LocalDirectory = "./models/pinned",
    RequiredFiles = ["model.onnx"],
    Revision = "main",
    ExpectedCommitSha = "1234567890abcdef1234567890abcdef12345678"
};

await downloader.DownloadFilesAsync(request);
Console.WriteLine(request.ResolvedCommitSha);

4) Check if files are already downloaded

bool ready = downloader.AreFilesAvailable(
    ["onnx/model.onnx", "tokenizer.json"],
    "./models/miniLM");

if (!ready)
{
    var missing = downloader.GetMissingFiles(
        ["onnx/model.onnx", "tokenizer.json"],
        "./models/miniLM");
    Console.WriteLine($"Missing {missing.Count} files");
}

5) Discover files in a repo before downloading

IReadOnlyList<RepoTreeEntry> entries = await downloader.ListRepoFilesAsync(
    "sentence-transformers/all-MiniLM-L6-v2",
    RepoType.Model);

foreach (var entry in entries.Where(e => !e.IsDirectory))
    Console.WriteLine($"{entry.Path} ({entry.SizeBytes} bytes)");

// Use the discovered paths directly as a download request
await downloader.DownloadFilesAsync(new DownloadRequest
{
    RepoId = "sentence-transformers/all-MiniLM-L6-v2",
    LocalDirectory = "./models/miniLM",
    RequiredFiles = [.. entries.Where(e => !e.IsDirectory).Select(e => e.Path)]
});

6) Track download progress

var progress = new Progress<DownloadProgress>(p =>
{
    if (p.Stage == DownloadStage.Downloading)
        Console.Write($"\r⬇️ [{p.CurrentFile}] {p.PercentComplete:F0}% (reused {ByteFormatHelper.FormatBytes(p.ResumedBytes)})");
    else
        Console.WriteLine($"{p.Stage}: {p.Message}");
});

await downloader.DownloadFilesAsync(new DownloadRequest
{
    RepoId = "sentence-transformers/all-MiniLM-L6-v2",
    LocalDirectory = "./models/miniLM",
    RequiredFiles = ["onnx/model.onnx", "tokenizer.json"],
    Progress = progress
});

7) Authentication (Private/Gated Repos)

Set the HF_TOKEN environment variable, or pass it explicitly:

var downloader = new HuggingFaceDownloader(new HuggingFaceDownloaderOptions
{
    AuthToken = "hf_your_token_here"
});

8) Use a private Hub endpoint

Set the HF_ENDPOINT environment variable, or configure the endpoint explicitly:

var downloader = new HuggingFaceDownloader(new HuggingFaceDownloaderOptions
{
    Endpoint = "https://huggingface.internal"
});

The explicit Endpoint takes precedence over HF_ENDPOINT; if neither is set, the downloader uses https://huggingface.co.

9) Dependency Injection

builder.Services.AddHuggingFaceDownloader(options =>
{
    options.Timeout = TimeSpan.FromMinutes(60);
});

// Then inject HuggingFaceDownloader in your services
public class MyModelService(HuggingFaceDownloader downloader)
{
    public async Task EnsureModelAsync()
    {
        await downloader.DownloadFilesAsync(new DownloadRequest
        {
            RepoId = "my-org/my-model",
            LocalDirectory = DefaultPathHelper.GetDefaultCacheDirectory("MyApp"),
            RequiredFiles = ["model.onnx", "tokenizer.json"]
        });
    }
}

9) Manage local cache programmatically

var cacheRoot = DefaultPathHelper.GetDefaultCacheDirectory("hfdownload");
var repoId = "sentence-transformers/all-MiniLM-L6-v2";

// List all cached repositories
var cachedRepos = downloader.ListCachedRepos(cacheRoot);

// Check whether required files are cached for one repo
bool isCached = downloader.IsCached(
    repoId,
    cacheRoot,
    ["onnx/model.onnx", "tokenizer.json"]);

// Get total cache size for one local repo directory
long size = downloader.GetCachedSize(
    Path.Combine(cacheRoot, DefaultPathHelper.SanitizeModelName(repoId)));

// Delete cached files for the repo (safe no-op if not found)
await downloader.DeleteCachedFilesAsync(repoId, cacheRoot);

Direct downloads also write ./models/.../.hf.download.resolved.json so callers can inspect the requested revision and resolved commit later.

Documentation

Related Projects

• ElBruno.PersonaPlex — Integrates this downloader to auto-download ONNX models for NVIDIA's PersonaPlex-7B-v1 full-duplex speech-to-speech model

Building from Source

git clone https://github.com/elbruno/ElBruno.HuggingFace.Downloader.git
cd ElBruno.HuggingFace.Downloader
dotnet build ElBruno.HuggingFace.Downloader.slnx
dotnet test ElBruno.HuggingFace.Downloader.slnx

Requirements

• .NET 8.0 SDK or later

🤝 Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📄 License

This project is licensed under the MIT License — see the LICENSE file for details.

👋 About the Author

Hi! I'm ElBruno 🧡, a passionate developer and content creator exploring AI, .NET, and modern development practices.

Made with ❤️ by ElBruno

If you like this project, consider following my work across platforms:

• 📻 Podcast: No Tienen Nombre — Spanish-language episodes on AI, development, and tech culture
• 💻 Blog: ElBruno.com — Deep dives on embeddings, RAG, .NET, and local AI
• 📺 YouTube: youtube.com/elbruno — Demos, tutorials, and live coding
• 🔗 LinkedIn: @elbruno — Professional updates and insights
• 𝕏 X: @elbruno — Quick tips, releases, and tech news