🦀 libmaxminddb-rs

An independent Rust implementation of the MaxMind DB (MMDB) v2 format. It reads IPv4/IPv6 databases with borrowed decoding and writes deterministic MMDB files.

---

📋 Table of Contents

• 🚀 Features
• ⚡ Architecture & Optimizations
• 📦 Installation & Cargo Features
• 🚀 Quickstart Guide
  • 🔍 1. Reading & Zero-Copy Lookups
  • Lookup API Reference
  • ✍️ 2. Database Creation & Serialization
  • 🔀 3. Merging Databases (Deep Merge)
• 📊 Benchmarks
  • ⚙️ Evaluated Libraries & Reproducibility Specification
  • 🏁 Reader Performance Summary — p99 Tail Latency & Peak Throughput
  • 📈 Visual Benchmark Charts
• 🔥 FlameGraph
• Documentation and compatibility
• Development and tests
• Code Coverage & Tests
• Contributing
• License

---

🚀 Features

• Full MMDB v2 Specification — Independent, pure-Rust implementation of the MaxMind DB v2 format for both IPv4 and IPv6 network lookups, without external C dependencies or wrapper overhead.
• Zero-Copy Deserialization — Derive-based deserialization (#[derive(MmdbDecode)]) that borrows string (&'a str) and binary (&'a [u8]) slices directly from the database buffer without heap allocations.
• Flexible Lookup APIs:
  • lookup_borrowed: Strongly typed, single-pass zero-copy decoding into user structs.
  • lookup_borrowed_opt: Borrowed decoding with an Option result for miss-heavy workloads.
  • lookup_borrowed_map: Borrowed decoding with a small callback result and None on a miss.
  • lookup_value: Generic dynamic inspection via borrowed ValueRef trees.
  • lookup_value_with_prefix: Longest-prefix matching returning both data and matched CIDR prefix length (subnet mask).
  • lookup: Owned deserialization through serde.
  • lookup_many: Order-preserving batch lookup with adaptive multithreading for bulk IP resolution.
  • lookup_exists: Check whether an address matches a record without decoding it.
• Database Creation & Writer — High-performance in-memory trie builder producing standard-compliant MMDB v2 binary files with automatic payload deduplication and optimal pointer widths (24, 28, or 32 bits).
• Multi-Source Dataset Merging — Built-in conflict resolution strategies for overlapping CIDR blocks (Replace, Append, AppendUnique, and recursive DeepMerge).
• Ergonomic Derive Macros — #[derive(MmdbDecode, MmdbEncode, MmdbRecord)] with #[mmdb(network)] attribute support for one-object database insertion and zero-boilerplate struct mapping.
• Flexible Storage Backends — Load databases from owned memory buffers (Reader::open), borrowed byte slices (Reader::from_bytes), or memory-mapped files (Reader::open_mmap).

---

⚡ Architecture & Optimizations

1. Zero-Copy & Allocation Avoidance

• Lifetime-Bound Borrowing: lookup_borrowed projects wire bytes straight into derived structs without intermediate ValueRef trees; string and byte fields borrow the reader buffer directly, avoiding String and Vec<u8> allocations.
• Compact Miss Path: Tree traversal represents misses without constructing a decoded record.
• Arena-Based Trie Builder: The writer stores trie nodes contiguously in a flat Vec<TrieNode> arena using u32 index references, eliminating individual heap-allocated pointer indirections (Box).
• Lower Writer Peak Memory: Replacing an existing prefix keeps the new shared Arc<Value> without cloning its payload, and finalization releases trie nodes before allocating the output buffer.

2. Search Tree & Traversal Optimizations

• Specialized Unaligned Node Traversal: Unaligned 64-bit word loads (load!) read each node in a single instruction, with dedicated branch-free decoding loops tailored for 24-bit, 28-bit, and 32-bit pointer layouts.
• Cache-Line Aligned Fast Tree: Opening a reader prepares a native-endian search tree aligned to 64-byte boundaries (AlignedNodes), packing 8 nodes per CPU cache line. Radix and byte-stride tables are built at the same time, so the first lookup does not construct an index. This trades open time and reader memory for lower steady-state lookup latency.
• Radix Accelerator Tables: Root-level lookup tables index the first 8, 16, or 20 bits of IPv4 and IPv6 prefixes, reducing dependent pointer-chasing iterations.
• CPU Cache Prefetching: Targeted _mm_prefetch instructions warm CPU cache lines ahead of sequential and multi-step tree traversals on supported architectures.

3. Vectorization & SIMD

• Hardware-Accelerated ASCII Probing: Vectorized SIMD scanners (AVX2 / SSE2 on x86_64, NEON on AArch64) validate ASCII strings in bulk before UTF-8 decoding.
• Out-of-Line Non-ASCII Fallback: Payloads with non-ASCII bytes divert to an out-of-line #[cold] validator, keeping standard library UTF-8 validation routines out of the hot instruction stream.

4. Concurrency & Throughput

• Shared Reader: Prepared tree indexes are initialized once and shared across lookup threads.
• Adaptive Batch Parallelism: lookup_many dynamically selects sequential execution for smaller workloads and chunked parallel processing via std::thread::scope for large batches ($\ge 4,096$ IPs).

5. Memory-Mapped File Access & Safety

• Direct Kernel Paging (open_mmap): Memory-maps database files to leverage OS page caches, reducing startup overhead and memory footprint across shared processes.
• Branch-Guarded Bounds Hardening: Unchecked slice accesses on hot paths are guarded by cheap, inlined range checks (in_range, cursor limits) and audited unsafe blocks with explicit safety invariants.

---

📦 Installation & Cargo Features

Add libmaxminddb-rs to your Cargo.toml:

[dependencies]
libmaxminddb-rs = "0.1"

⚙️ Cargo Features

The fast tree is built into the reader and is always prepared during Reader::open, Reader::open_mmap, Reader::from_vec, or Reader::from_bytes for every valid MMDB record size. It is no longer a Cargo feature. Common 24/28/32-bit records use aligned u32 nodes and acceleration tables; 36–64-bit records use decoded u64 children (16 bytes per node). Preparation errors are returned when opening the database.

---

🚀 Quickstart Guide

🔍 1. Reading & Zero-Copy Lookups

This complete example builds a small database, borrows a typed record from it, and handles an absent address. &str fields refer to the reader's MMDB bytes; the record cannot outlive the reader.

use libmaxminddb_rs::{Error, MetadataBuilder, MmdbDecode, MmdbEncode, Reader, Writer};
use std::net::IpAddr;

#[derive(MmdbDecode, MmdbEncode)]
struct NetworkRecord<'a> {
    asn: u32,
    org: &'a str,
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let metadata = MetadataBuilder::new().ip_version(4).build()?;
    let mut writer = Writer::with_metadata(metadata);
    writer.insert_encoded(
        "198.51.100.0/24".parse()?,
        &NetworkRecord { asn: 64512, org: "Example Network" },
    )?;
    let bytes = writer.finish()?;
    let reader = Reader::from_bytes(&bytes)?;

    let ip: IpAddr = "198.51.100.7".parse()?;
    let record: NetworkRecord<'_> = reader.lookup_borrowed(ip)?;
    assert_eq!(record.asn, 64512);
    assert_eq!(record.org, "Example Network");

    // Project a large decoded record to a small return value on a hit.
    let asn = reader.lookup_borrowed_map(ip, |r: NetworkRecord<'_>| r.asn)?;
    assert_eq!(asn, Some(64512));

    let missing: IpAddr = "203.0.113.7".parse()?;
    assert!(matches!(reader.lookup_borrowed::<NetworkRecord<'_>>(missing), Err(Error::NotFound)));
    Ok(())
}

For a zero-copy walkthrough that can be run with Cargo, see examples/zero_copy_lookup.rs.

Lookup API Reference

The example below covers every public Reader lookup method. Replace the MMDB path and NetworkRecord fields with your database's schema. The lookup example also uses serde_json::Value, so applications using that line need a serde_json dependency. Borrowed strings remain valid only while the reader is alive.

use std::error::Error;
use std::net::IpAddr;
use libmaxminddb_rs::{MmdbDecode, Reader, ValueRef};

#[derive(MmdbDecode)]
struct NetworkRecord<'a> {
    asn: u32,
    org: &'a str,
}

fn main() -> Result<(), Box<dyn Error>> {
    let reader = Reader::open("your-database.mmdb")?;
    let ip: IpAddr = "2001:db8::1".parse()?;

    // lookup_borrowed decodes typed fields; strings borrow the MMDB buffer.
    let record: NetworkRecord<'_> = reader.lookup_borrowed(ip)?;
    println!("AS{}: {}", record.asn, record.org);

    // lookup_borrowed_opt returns None on a miss or a decode failure.
    let optional: Option<NetworkRecord<'_>> = reader.lookup_borrowed_opt(ip);
    println!("Typed record found: {}", optional.is_some());

    // lookup_borrowed_map returns a small result without returning the full record.
    // It returns Ok(None) on a miss and preserves decode errors.
    let asn: Option<u32> = reader.lookup_borrowed_map(ip, |r: NetworkRecord<'_>| r.asn)?;
    println!("ASN: {asn:?}");

    // lookup_value returns a generic borrowed value; maps and arrays allocate containers.
    let value: ValueRef<'_> = reader.lookup_value(ip)?;
    println!("Generic value: {value:?}");

    // lookup_value_with_prefix also reports the matched network prefix length.
    let (_value, prefix): (ValueRef<'_>, u8) = reader.lookup_value_with_prefix(ip)?;
    println!("Matched prefix: /{prefix}");

    // lookup converts the record through serde into an owned JSON value.
    let owned: serde_json::Value = reader.lookup(ip)?;
    println!("Owned value: {owned}");

    // lookup_many returns one ordered Result per address and may use worker threads.
    let addresses = [ip, "2001:db8::2".parse()?];
    let results: Vec<_> = reader.lookup_many(&addresses);
    println!("Batch length: {}", results.len());

    // lookup_exists answers whether an address has a record.
    println!("Record exists: {}", reader.lookup_exists(ip));

    Ok(())
}

✍️ 2. Database Creation & Serialization

The writer constructs standard-compliant MMDB v2 binary databases in memory with automatic payload deduplication and optimal pointer sizing (24, 28, or 32 bits).

The #[derive(MmdbEncode, MmdbRecord)] derive macros provide a seamless one-object insertion API: the #[mmdb(network)] attribute marks the CIDR subnet key and automatically omits it from the serialized payload data.

(Run this complete example with cargo run --example custom_record)

use std::error::Error;
use libmaxminddb_rs::{IpNetwork, MetadataBuilder, MmdbEncode, MmdbRecord, Writer};

/// Custom record deriving both MmdbEncode and MmdbRecord.
/// The #[mmdb(network)] field designates the subnet without storing it in the payload.
#[derive(Debug, MmdbEncode, MmdbRecord)]
struct SecurityEntry<'a> {
    #[mmdb(network)]
    network: IpNetwork,
    country: &'a str,
    threat_score: u32,
    is_tor_exit: bool,
}

fn main() -> Result<(), Box<dyn Error>> {
    // 1. Configure database metadata (database type, IP version, languages, descriptions)
    let metadata = MetadataBuilder::new()
        .database_type("Security-Intelligence")
        .ip_version(4)
        .description("en", "IP Threat Intelligence Feed")
        .build()?;

    let mut writer = Writer::with_metadata(metadata);

    // 2. Insert records using the ergonomic one-object insertion API
    writer.insert_entry(&SecurityEntry {
        network: "203.0.113.0/24".parse()?,
        country: "FR",
        threat_score: 85,
        is_tor_exit: false,
    })?;

    writer.insert_entry(&SecurityEntry {
        network: "198.51.100.128/25".parse()?,
        country: "US",
        threat_score: 95,
        is_tor_exit: true,
    })?;

    // 3. Finalize into a contiguous in-memory byte buffer
    let mmdb_bytes: Vec<u8> = writer.finish()?;
    println!("Database serialized successfully: {} bytes", mmdb_bytes.len());

    // Alternatively, serialize directly to a file on disk:
    // writer.write_to_file("threat-intelligence.mmdb")?;

    Ok(())
}

🔀 3. Merging Databases (Deep Merge)

A common challenge in network telemetry is combining multi-source intelligence on identical or overlapping subnets — such as augmenting a baseline IP geolocation database with an external real-time threat intelligence feed.

By default, MMDB writers use MergeStrategy::Replace, where inserting an existing subnet completely wipes out the previous record, destroying any previously associated geographic coordinates or ISP metadata.

Configuring the writer with MergeStrategy::DeepMerge enables recursive hierarchical merging:

• Nested Maps: Recursively merged key-by-key. Keys present only in the base record are preserved (country, city, details.timezone). New keys are seamlessly added (details.datacenter, security). Conflicting nested scalar values are updated with the latest value (details.accuracy_radius 50 $\to$ 10).
• Arrays: Concatenated in order (["residential", "broadband"] $+$ ["vpn_exit_node"]).
• Scalars: Conflicting scalar values are replaced by the newest record.

(Run this complete example with cargo run --example deep_merge)

use std::error::Error;
use std::net::IpAddr;
use libmaxminddb_rs::{MergeStrategy, MetadataBuilder, Reader, Writer};

fn main() -> Result<(), Box<dyn Error>> {
    // 1. Initialize Writer configured with recursive DeepMerge strategy
    let metadata = MetadataBuilder::new()
        .database_type("Enriched-GeoIP-Threat")
        .ip_version(4)
        .build()?;

    let mut writer = Writer::with_metadata(metadata)
        .merge_strategy(MergeStrategy::DeepMerge);

    let target_subnet = "203.0.113.0/24".parse()?;

    // 2. Base geolocation feed: general geographic coordinates and ISP tags
    let base_geo_record = serde_json::json!({
        "country": "FR",
        "city": "Paris",
        "details": {
            "timezone": "Europe/Paris",
            "accuracy_radius": 50
        },
        "network_tags": ["residential", "broadband"]
    });
    writer.insert(target_subnet, &base_geo_record)?;

    // 3. Threat intelligence feed: enriches the same subnet with security telemetry
    let threat_intel_record = serde_json::json!({
        "details": {
            "accuracy_radius": 10,       // overrides conflicting scalar with higher precision
            "datacenter": "PAR-01"       // adds a new nested key into "details"
        },
        "network_tags": ["vpn_exit_node"], // appends new element to the existing array
        "security": {                    // adds an entirely new top-level nested map
            "is_proxy": true,
            "threat_score": 85
        }
    });
    writer.insert(target_subnet, &threat_intel_record)?;

    // 4. Finalize database and query with zero-copy Reader
    let mmdb_bytes = writer.finish()?;
    let reader = Reader::from_bytes(&mmdb_bytes)?;

    let target_ip: IpAddr = "203.0.113.42".parse()?;
    let (merged_value, prefix_len) = reader.lookup_value_with_prefix(target_ip)?;

    println!("Matched subnet prefix: /{prefix_len}");
    println!("{}", serde_json::to_string_pretty(&merged_value.to_json())?);

    Ok(())
}

🔍 Verified Output

Executing the lookup yields the fully unified document combining both datasets:

{
  "city": "Paris",
  "country": "FR",
  "details": {
    "accuracy_radius": 10,
    "datacenter": "PAR-01",
    "timezone": "Europe/Paris"
  },
  "network_tags": [
    "residential",
    "broadband",
    "vpn_exit_node"
  ],
  "security": {
    "is_proxy": true,
    "threat_score": 85
  }
}

📋 Merge Strategy Comparison

---

📊 Benchmarks

Reproducible cross-library benchmark suite comparing libmaxminddb-rs against industry standard implementations in Rust, C, and Go.

⚙️ Evaluated Libraries & Reproducibility Specification

Environment: Linux x86_64 · AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics · rustc 1.90.0 · Deterministic SplitMix64 datasets with pre-allocated memory.

Execute all benchmarks and regenerate reports with a single command:

make bench-compare

To run the same comparative suite with pinned Rust and Go toolchains in Docker, use make bench-compare-docker (Docker with Compose required). It selects the system default Docker context, even if Docker Desktop is the current CLI context; set BENCH_DOCKER_CONTEXT=name to choose another daemon. The command builds the image, runs make bench-compare in a temporary container, and prints the host paths of the generated HTML report, SVG charts, JSON/CSV results, and updated README.md when it finishes. Docker results and build caches remain under target/docker-bench/. Compare measurements only from compatible host CPUs and Docker resource limits.

Both benchmark commands generate an interactive HTML report at benchmark-report/index.html with all measured results, charts, and sortable tables. Open this local file after the run.

Full HTML report preview (static image)

Database-size and writer benchmarks use 1K, 10K, 100K, 500K, 1M, 1.5M, 2M, 5M entries, with the same fixed seed, query workload and batch settings at every size.

Reader RSS is also compared at these eight sizes for the four Rust/C libraries, using identical databases and queries, mmap, and three isolated processes per point. The generated HTML report's Memory section shows RSS after open and the lookup peak, with exact hover values and explicit unavailable measurements. See the memory protocol for reproduction and interpretation.

🏁 Reader Performance Summary — p99 Tail Latency & Peak Throughput

Measured on AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics under Linux:

📈 Visual Benchmark Charts

Candlestick Percentile Rank — IPv4 Lookups

Candlestick Percentile Rank — IPv6 Lookups

IPv4 Throughput — Random Lookups (1M)

IPv6 Throughput — Random Lookups (1M)

IPv4 Throughput — Absent Keys (1M)

IPv6 Throughput — Absent Keys (1M)

Peak Concurrent Throughput — 16 Threads

Peak Concurrent Throughput — 16 Threads (IPv6)

IPv4 Random Lookup — Worker Scaling by API

IPv6 Random Lookup — Worker Scaling by API

p99 Tail Latency vs Database Size

Peak RSS During Lookups (mmap)

🔥 FlameGraph

Profile all public lookup methods on IPv4 and IPv6 and regenerate the SVG with:

make flamegraph

The command writes the lookup-only profile to target/flamegraph/lookup-flamegraph.svg and updates the image below. The graph covers lookup_borrowed, lookup_borrowed_opt, lookup_borrowed_map, lookup_value, lookup_value_with_prefix, lookup, lookup_many, and lookup_exists.

Documentation and compatibility

The Rust API documentation describes the reader, writer, value types, and derive macros. Runnable usage examples are in examples/. The crate supports MMDB v2 files, IPv4 and IPv6, and 24-, 28-, or 32-bit tree records. Its minimum supported Rust version is 1.90 (edition 2024). Reader::open_mmap is unsafe because callers must keep the mapped file unchanged while the reader exists.

For implementation details and performance protocols, see docs/ and AGENTS.md. The derive proc-macro crate is a separate package and must be published before the main crate.

Development and tests

cargo build --all-features
cargo fmt --all -- --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo doc --no-deps --all-features
make bench-performance
make bench-writer

make publish-check checks formatting, linting, tests, documentation, and both package archives. make publish uploads the proc-macro crate and then the main crate; make release tags a clean commit and creates a GitHub Release. Run those commands only when you intend to publish. The contribution guide explains the development workflow.

Code Coverage & Tests

Run the complete test suite with the Makefile target. It includes the workspace (including derive), documentation tests, feature-isolated builds, and the benchmark tooling crates:

make test-all

Install cargo-llvm-cov, then generate the HTML report and per-file summary with:

make code-coverage

Open target/coverage/html/index.html in a browser. Both Makefile targets refresh this section and the test and code coverage badges; make test-all runs coverage after its other tests. Documentation tests are run separately and are not included in these coverage figures because instrumenting them requires a nightly Rust toolchain.

The latest local coverage run measured 97.48% overall line coverage and 95.92% region coverage.

Code coverage by file (line, region, and function coverage)

File	Lines	Regions	Functions
derive/src/lib.rs	99.61% (253/254)	99.75% (403/404)	100.00% (37/37)
src/decoder/ascii.rs	97.35% (147/151)	98.35% (298/303)	100.00% (12/12)
src/decoder/mod.rs	96.83% (519/536)	95.03% (898/945)	96.55% (28/29)
src/decoder/raw.rs	98.09% (513/523)	95.06% (1001/1053)	94.00% (47/50)
src/encoder.rs	100.00% (112/112)	94.32% (216/229)	100.00% (8/8)
src/metadata.rs	96.58% (226/234)	96.06% (341/355)	100.00% (26/26)
src/reader/marker.rs	95.48% (211/221)	95.89% (420/438)	100.00% (18/18)
src/reader/mod.rs	97.50% (781/801)	94.81% (1408/1485)	98.53% (67/68)
src/reader/tree.rs	96.31% (835/867)	95.69% (1577/1648)	100.00% (54/54)
src/traits.rs	100.00% (354/354)	97.83% (631/645)	100.00% (59/59)
src/value.rs	98.67% (593/601)	97.78% (750/767)	98.15% (106/108)
src/writer/mod.rs	96.64% (863/893)	95.35% (1416/1485)	95.24% (60/63)
Total	97.48% (5407/5547)	95.92% (9359/9757)	98.12% (522/532)

Contributing

Issues and pull requests are welcome. Please read CONTRIBUTING.md before changing public APIs, parser invariants, or hot paths. User-visible changes should be recorded in CHANGELOG.md.

License

Licensed under either MIT or Apache-2.0, at your option.