kunpengcompute/malloc-bench

463 commits

updated Jul 1, 2026

README

Mimalloc-bench

Suite for benchmarking malloc implementations, originally developed for benchmarking mimalloc. Collection of various benchmarks from the academic literature, together with automated scripts to pull specific versions of benchmark programs and allocators from Github and build them.

Due to the large variance in programs and allocators, the suite is currently only developed for Unix-like systems, and specifically Ubuntu with apt-get, Fedora with dnf, and macOS (for a limited set of allocators and benchmarks). The only system-installed allocator used is glibc's implementation that ships as part of Linux's libc. All other allocators are downloaded and built as part of build-bench-env.sh -- if you are looking to run these benchmarks on a different Linux distribution look at the setup_packages function to see the packages required to build the full set of allocators.

It is quite easy to add new benchmarks and allocator implementations -- please do so!.

Enjoy, Daan

Note that all the code in the bench directory is not part of mimalloc-bench as such, and all programs in the bench directory are governed under their own specific licenses and copyrights as detailed in their README.md (or license.txt) files. They are just included here for convenience.

Benchmarking

The build-bench-env.sh script with the all argument will automatically pull all needed benchmarks and allocators and build them in the extern directory:

~/dev/mimalloc-bench> ./build-bench-env.sh all

It starts installing packages and you will need to enter the sudo password. All other programs are build in the mimalloc-bench/extern directory. Use ./build-bench-env.sh -h to see all options.

If everything succeeded, you can run the full benchmark suite (from out/bench) as:

~/dev/mimalloc-bench> cd out/bench
~/dev/mimalloc-bench/out/bench>../../bench.sh alla allt

Or just test mimalloc and tcmalloc on cfrac and larson with 16 threads:

~/dev/mimalloc-bench/out/bench>../../bench.sh --procs=16 mi tc cfrac larson

Or run the quick essential benchmark set (quickt) and append the matching KQ benchmark group explicitly:

~/dev/mimalloc-bench/out/bench>../../bench.sh --procs=\nproc` -r=5 -n=1 -s=1 kq_09 je tc mi quickt kq_bench_09`

Generally, you can specify the allocators (mi, je, tc, hd, sys (system allocator)) etc, and the benchmarks , cfrac, espresso, barnes, lean, larson, alloc-test, cscratch, etc. Or all allocators (alla) and tests (allt). Use --procs=<n> to set the concurrency, and use --help to see all supported allocators and benchmarks.

Current Allocators

Supported allocators are as follow, see build-bench-env.sh for the versions:

dieharder: The DieHarder allocator is an error-resistant memory allocator for Windows, Linux, and Mac OS X.
ff: ffmalloc, from the Usenix Security 21 paper
fg: The FreeGuard allocator, from the CCS 17 paper
gd: The Guarder allocator is a tunable secure allocator by the UTSA.
hd: The Hoard allocator by Emery Berger [1]. This is one of the first multi-thread scalable allocators.
hm: The Hardened Malloc from GrapheneOS, security-focused.
iso: The Isoalloc allocator, isolation-based aiming at providing a reasonable level of security without sacrificing too much the performances.
je: The jemalloc allocator by Jason Evans, now developed at Facebook and widely used in practice, for example in FreeBSD and Firefox.
lf: The lockfree-malloc allocator, multi-thread scalability-focused.
lp: The libpas allocator, used by WebKit.
lt: The ltalloc allocator, a multi-threaded memory allocator based on free lists best suited for many small allocations.
mesh: The mesh allocator, a memory allocator that automatically reduces the memory footprint of C/C++ applications. Also tested as nomesh with the meshing feature disabled.
mi: The mimalloc allocator. We can also test the debug version as dmi (this can be used to check for any bugs in the benchmarks), and the secure version as smi.
mng: musl's memory allocator.
pa: The PartitionAlloc allocator used in Chromium.
rp: The rpmalloc allocator uses 16-byte aligned allocations and is developed by Mattias Jansson at Epic Games, used for example in Haiku.
sc: The scalloc allocator, a fast, multicore-scalable, low-fragmentation memory allocator
scudo: The scudo allocator used by Fuschia and Android.
sg: The slimguard allocator, designed to be secure and memory-efficient.
sm: The Supermalloc allocator by Bradley Kuszmaul uses hardware transactional memory to speed up parallel operations.
sn: The snmalloc allocator is a recent concurrent message passing allocator by Liétar et al. [8].
tbb: The Intel TBB allocator that comes with the Thread Building Blocks (TBB) library [7].
tc: The tcmalloc allocator which comes as part of the Google performance tools, now maintained by the commuity.
tcg: The tcmalloc allocator, maintained and used by Google.
yal: The yalloc yet another allocator aims at balancing safety and compactness.
sys: The system allocator. Here we usually use the glibc allocator (which is originally based on Ptmalloc2).

Current Benchmarks

The first set of benchmarks are real world programs, or are trying to mimic some, and consists of:

barnes: a hierarchical n-body particle solver [4], simulating the gravitational forces between 163840 particles. It uses relatively few allocations compared to cfrac and espresso but is multithreaded.
cfrac: by Dave Barrett, implementation of continued fraction factorization, using many small short-lived allocations.
espresso: a programmable logic array analyzer, described by Grunwald, Zorn, and Henderson [3]. in the context of cache aware memory allocation.
gs: have ghostscript process the entire Intel Software Developer’s Manual PDF, which is around 5000 pages.
leanN: The Lean compiler by de Moura et al, version 3.4.1, compiling its own standard library concurrently using N threads (./lean --make -j N). Big real-world workload with intensive allocations.
redis: running redis-benchmark, with 1 million requests pushing 10 new list elements and then requesting the head 10 elements, and measures the requests handled per second. Simulates a real-world workload.
larsonN: by Larson and Krishnan [2]. Simulates a server workload using 100 separate threads which each allocate and free many objects but leave some objects to be freed by other threads. Larson and Krishnan observe this behavior (which they call bleeding) in actual server applications, and the benchmark simulates this.
larsonN-sized: same as the larsonN except it uses sized deallocation calls which have a fast path in some allocators.
lua: compiling the lua interpreter.
z3: perform some computations in z3.

The second set of benchmarks are stress tests and consist of:

alloc-test: a modern allocator test developed by OLogN Technologies AG (ITHare.com) Simulates intensive allocation workloads with a Pareto size distribution. The alloc-testN benchmark runs on N cores doing 100·10⁶ allocations per thread with objects up to 1KiB in size. Using commit 94f6cb (master, 2018-07-04)
cache-scratch: by Emery Berger [1]. Introduced with the Hoard allocator to test for passive-false sharing of cache lines: first some small objects are allocated and given to each thread; the threads free that object and allocate immediately another one, and access that repeatedly. If an allocator allocates objects from different threads close to each other this will lead to cache-line contention.
cache_trash: part of Hoard benchmarking suite, designed to exercise heap cache locality.
glibc-simple and glibc-thread: benchmarks for the glibc.
kq_bench_08/09/12: the malloc_benchmark benchmark from boostkit/kqmalloc v0.19.0-beta0, built as extern/kq_bench with make clean kunpeng. These group aliases expand to kq_bench_<hip>_m0, kq_bench_<hip>_m1, and kq_bench_<hip>_m2; each item runs one mode with -threads N -affinity 0.
malloc-large: part of mimalloc benchmarking suite, designed to exercice large (several MiB) allocations.
mleak: check that terminate threads don't "leak" memory.
rptest: modified version of the rpmalloc-benchmark suite.
mstress: simulates real-world server-like allocation patterns, using N threads with with allocations in powers of 2 where objects can migrate between threads and some have long life times. Not all threads have equal workloads and after each phase all threads are destroyed and new threads created where some objects survive between phases.
rbstress: modified version of allocator_bench, allocates chunks in memory via ruby shenanigans.
sh6bench: by MicroQuill as part of SmartHeap. Stress test where some of the objects are freed in a usual last-allocated, first-freed (LIFO) order, but others are freed in reverse order. Using the public source (retrieved 2019-01-02)
sh8benchN: by MicroQuill as part of SmartHeap. Stress test for multi-threaded allocation (with N threads) where, just as in larson, some objects are freed by other threads, and some objects freed in reverse (as in sh6bench). Using the public source (retrieved 2019-01-02)
xmalloc-testN: by Lever and Boreham [5] and Christian Eder. We use the updated version from the SuperMalloc repository. This is a more extreme version of the larson benchmark with 100 purely allocating threads, and 100 purely deallocating threads with objects of various sizes migrating between them. This asymmetric producer/consumer pattern is usually difficult to handle by allocators with thread-local caches.
recommend-like: jemalloc-specific KPI bench simulating search/recommendation request workloads — short-lived objects, high churn, steady-state dirty pages. Reads 5 jemalloc stats via mallctl (Allocated/RSS, Dirty/RSS, Metadata/Allocated, ChurnRate, 16-32K alloc count) and validates whether the bench shape matches real-service profiles. Three entry points: run_sanity.sh (single-run KPI PASS/FAIL), run.sh (B0 vs A multi-round compare with MAD/median noise floor), and the main bench.sh recommend-like driver — see bench/recommend-like/README.md. Jemalloc-only: the bench binary links jemalloc at compile time for the stats interface, so bench.sh filters alloc_run against a jemalloc-family whitelist (je myje myje-base, overridable via RECOMMEND_LIKE_ALLOCS) and warns + skips on other allocators to avoid writing meaningless KPI rows into benchres.csv. Example: ../../bench.sh je recommend-like (OK), ../../bench.sh tc mi recommend-like (skipped).

Finally, there is a security benchmark aiming at checking basic security properties of allocators.

Benchmark Evaluation Dimensions

The full benchmark suite covers the following key evaluation dimensions:

Dimension	Benchmarks
Single-thread throughput	cfrac, espresso, glibc-simple, alloc-test1
Multi-thread scalability	leanN, alloc-testN, sh8bench, xmalloc-testN, glibc-thread
Cross-thread object migration	larson, sh8bench, xmalloc-test, mstress
Cache locality / false sharing	cache-scratch, cache-thrash, espresso
Large allocation efficiency	malloc-large
Memory footprint / fragmentation	all benchmarks (RSS peak recorded)
Thread-exit memory reclamation	mleak
Real-world application workloads	redis, gs, lean, lua, linux, rocksdb, z3, spec
Security properties	security (double-free, overflow, underflow, use-after-free, …)

Benchmark Ranking: Generality and Discriminative Power

The table below ranks each benchmark by two criteria:

Generality — how broadly representative the workload is of real-world usage
Effectiveness — how well the benchmark differentiates allocator quality

Rank	Benchmark	Generality	Effectiveness	Rationale
1	leanN	★★★★★	★★★★★	Large real-world multi-threaded compiler workload with intensive allocations; best at exposing practical performance differences
2	alloc-testN	★★★★☆	★★★★★	Pareto-distributed sizes mimic realistic programs; 100M allocs/thread sharply differentiates throughput
3	larson / larson-sized	★★★★★	★★★★☆	Classic server cross-thread "bleeding" pattern; used in nearly every allocator paper
4	cfrac	★★★★☆	★★★★☆	Canonical single-thread small-object benchmark; clearly exposes single-threaded allocation efficiency
5	xmalloc-testN	★★★☆☆	★★★★★	Extreme producer/consumer asymmetry; most effective at exposing weaknesses in thread-local caching strategies
6	cache-scratch	★★★☆☆	★★★★★	Precisely measures passive false sharing; the only test that effectively detects this critical issue
7	redis	★★★★★	★★★☆☆	High real-world relevance (server workload), but allocator differences are typically small
8	mstress	★★★★☆	★★★★☆	Multi-phase thread recreation with object migration; good all-round stress test
9	espresso	★★★★☆	★★★☆☆	Classic cache-locality benchmark, but single-threaded nature limits discriminative power
10	rptest	★★★☆☆	★★★★☆	Highly configurable; covers multiple dimensions, but use-case is narrower
11	linux	★★★★★	★★★☆☆	Most realistic build workload; allocator impact is often overshadowed by I/O
12	rocksdb	★★★★☆	★★★☆☆	Good database-workload coverage; bottleneck is often I/O rather than allocation
13	sh6bench / sh8bench	★★★☆☆	★★★☆☆	Classic stress tests with synthetic free-order patterns
14	malloc-large	★★★☆☆	★★★☆☆	Specific but important large-allocation scenario
15	barnes	★★★☆☆	★★☆☆☆	Too few allocations to meaningfully differentiate allocators
16	security (135 tests)	★★★★☆	★★★★★	Unique safety dimension; measures security properties rather than performance
17	mleak	★★☆☆☆	★★★★☆	Narrow scenario, but the thread-exit leak issue it detects is critical
18	spec	★★★★★	★★★★☆	Industry-standard suite; requires a separate SPEC CPU 2017 installation
19	glibc-simple / glibc-thread	★★★☆☆	★★☆☆☆	Simple glibc-derived patterns with limited discriminative power
20	z3, gs, lua, lean-mathlib, rbstress, cache-thrash	★★☆☆☆	★★☆☆☆	Supplementary tests covering long-tail scenarios

Quick Benchmark Set (`quickt`)

For a fast but representative evaluation, use the quickt option and append the matching KQ benchmark group explicitly:

~/dev/mimalloc-bench/out/bench> ../../bench.sh --procs=`nproc` -r=5 -n=1 -s=1 kq_09 je tc mi quickt kq_bench_09

This runs carefully selected benchmarks that together cover the most important and distinct evaluation dimensions:

Benchmark	Key Dimension	Why Selected
cfrac	Single-thread small-object throughput	The canonical baseline for pure allocation speed; fast and low-noise
alloc-testN	Multi-thread scalability	Pareto size distribution; 100M allocs/thread yields the clearest throughput signal
larson-sized	Cross-thread object migration + sized deallocation	Represents server workloads where objects are freed by threads other than their allocator; also exercises the sized-free fast path
leanN	Real-world dense multi-thread allocation	Large-scale compiler workload; most predictive of real application performance
redis	Real-world single-threaded workload	High real-world relevance (server workload)
rocksdb	Real-world storage engine workload	Good database-workload coverage; bottleneck is often I/O rather than allocation; but particularly effective for validating memory usage characteristics
kq_bench_08/09/12	KQ malloc stress modes	Optional group aliases that run the kqmalloc `malloc_benchmark` modes using explicit HIP08, HIP09, or HIP12 binaries

These benchmarks cover single-threaded performance, multi-threaded scalability, cross-thread migration, real-world behavior, and memory usage - together giving a meaningful and well-rounded picture of allocator quality.

For a more thorough evaluation, add xmalloc-test, mstress, cache-scratch, malloc-large, and security to cover extreme cross-thread patterns, large allocations, false-sharing detection, and security properties.

One-shot runner: `run_quickt.sh`

run_quickt.sh is a convenience wrapper that runs the quickt set, appends the CPU-matched kq_bench group, and immediately feeds the results through scripts/analyze_bench.py (median, MAD, and ratio tables appended to benchres.csv).

# Default: auto-detects KQ allocator and kq_bench variants from CPU part, plus je tc mi
./run_quickt.sh

# Explicit allocator list
./run_quickt.sh kq_09 je tc mi

# Override rounds and core count
ROUNDS=3 PROCS=4 ./run_quickt.sh kq_09 je tc mi

The script auto-detects the KQ allocator and matching kq_bench variant from /proc/cpuinfo:

CPU part	Allocator	kq_bench
`0xd06`	`kq_12`	`kq_bench_12`
`0xd02`	`kq_09`	`kq_bench_09`
other	`kq_08`	`kq_bench_08`

After the run, scripts/analyze_bench.py prints median, MAD, and ratio tables to stdout and appends them to out/bench/benchres.csv.

Example

Below is an example (Apr 2019) of the benchmark results on an HP Z4-G4 workstation with a 4-core Intel® Xeon® W2123 at 3.6 GHz with 16GB ECC memory, running Ubuntu 18.04.1 with LibC 2.27 and GCC 7.3.0.

bench-z4-1 bench-z4-2

Memory usage:

(note: the xmalloc-testN memory usage should be disregarded is it allocates more the faster the program runs. Unfortunately, there are no entries for SuperMalloc in the leanN and xmalloc-testN benchmarks as it faulted on those)

Results and notable usages

Improvements

Minor performances improvement in isoalloc
Parallel compilation support in DieHarder
Portability improvement in Intel TBB malloc
Various portability improvements in Google's tcmalloc
Improved double-free detection in snmalloc
Fixed compilation on modern glibc in SlimGuard
A crash in isoalloc
Caught a compilation issue in rpmalloc
Portability issues in rpmalloc

Notable usages and papers

Provided data for the glib allocator.
Provided data for snmalloc hardening.
Used as main benchmark suite by S2malloc: Statistically Secure Allocator for Use-After-Free Protection And More by Ruizhe Wang, Meng Xu and N. Asokan
Used as benchmark suite by StarMalloc: Verifying a Modern, Hardened Memory Allocator by Antonin Reitz, Aymeric Fromherz, and Jonathan Protzenko

References

[1] Emery D. Berger, Kathryn S. McKinley, Robert D. Blumofe, and Paul R. Wilson. Hoard: A Scalable Memory Allocator for Multithreaded Applications the Ninth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS-IX). Cambridge, MA, November 2000. pdf
[2] P. Larson and M. Krishnan. Memory allocation for long-running server applications. In ISMM, Vancouver, B.C., Canada, 1998. pdf
[3] D. Grunwald, B. Zorn, and R. Henderson. Improving the cache locality of memory allocation. In R. Cartwright, editor, Proceedings of the Conference on Programming Language Design and Implementation, pages 177–186, New York, NY, USA, June 1993. pdf
[4] J. Barnes and P. Hut. A hierarchical O(n*log(n)) force-calculation algorithm. Nature, 324:446-449, 1986.
[5] C. Lever, and D. Boreham. Malloc() Performance in a Multithreaded Linux Environment. In USENIX Annual Technical Conference, Freenix Session. San Diego, CA. Jun. 2000. Available at https://github.com/kuszmaul/SuperMalloc/tree/master/tests
[6] Timothy Crundal. Reducing Active-False Sharing in TCMalloc. 2016. http://courses.cecs.anu.edu.au/courses/CSPROJECTS/16S1/Reports/Timothy*Crundal*Report.pdf. CS16S1 project at the Australian National University.
[7] Alexey Kukanov, and Michael J Voss. The Foundations for Scalable Multi-Core Software in Intel Threading Building Blocks. Intel Technology Journal 11 (4). 2007
[8] Paul Liétar, Theodore Butler, Sylvan Clebsch, Sophia Drossopoulou, Juliana Franco, Matthew J Parkinson, Alex Shamis, Christoph M Wintersteiger, and David Chisnall. Snmalloc: A Message Passing Allocator. In Proceedings of the 2019 ACM SIGPLAN International Symposium on Memory Management, 122–135. ACM. 2019.

Contributors

215 commits

169 commits

25 commits

13 commits

Similar repos

daanx/mimalloc-bench

Suite for benchmarking malloc implementations.

516

microsoft/mimalloc

mimalloc is a compact general purpose allocator with excellent performance.

13,392

emeryberger/Hoard

The Hoard Memory Allocator: A Fast, Scalable, and Memory-efficient Malloc for Linux, Windows, and…

1,250

mjansson/rpmalloc-benchmark

Benchmarks for our memory allocator

mjansson/rpmalloc

Public domain cross platform lock free thread caching 16-byte aligned memory allocator implemented…

2,508

jemalloc/jemalloc

11,190

bloomberg/bde-allocator-benchmarks

A set of benchmarking tools used to quantify the performance of BDE-style polymorphic allocators.

evanwashere/mitata

benchmark tooling that loves you ❤️

2,534

Languages

81.4%

Shell

7.4%

C++

7.0%

Python

3.4%

kunpengcompute/malloc-bench

463 commits

updated Jul 1, 2026

See the code

README

Mimalloc-bench

It is quite easy to add new benchmarks and allocator implementations -- please do so!.

Enjoy, Daan

Benchmarking

The build-bench-env.sh script with the all argument will automatically pull all needed benchmarks and allocators and build them in the extern directory:

~/dev/mimalloc-bench> ./build-bench-env.sh all

It starts installing packages and you will need to enter the sudo password. All other programs are build in the mimalloc-bench/extern directory. Use ./build-bench-env.sh -h to see all options.

If everything succeeded, you can run the full benchmark suite (from out/bench) as:

~/dev/mimalloc-bench> cd out/bench
~/dev/mimalloc-bench/out/bench>../../bench.sh alla allt

Or just test mimalloc and tcmalloc on cfrac and larson with 16 threads:

~/dev/mimalloc-bench/out/bench>../../bench.sh --procs=16 mi tc cfrac larson

Or run the quick essential benchmark set (quickt) and append the matching KQ benchmark group explicitly:

~/dev/mimalloc-bench/out/bench>../../bench.sh --procs=\nproc` -r=5 -n=1 -s=1 kq_09 je tc mi quickt kq_bench_09`

Current Allocators

Supported allocators are as follow, see build-bench-env.sh for the versions:

dieharder: The DieHarder allocator is an error-resistant memory allocator for Windows, Linux, and Mac OS X.
ff: ffmalloc, from the Usenix Security 21 paper
fg: The FreeGuard allocator, from the CCS 17 paper
gd: The Guarder allocator is a tunable secure allocator by the UTSA.
hd: The Hoard allocator by Emery Berger [1]. This is one of the first multi-thread scalable allocators.
hm: The Hardened Malloc from GrapheneOS, security-focused.
iso: The Isoalloc allocator, isolation-based aiming at providing a reasonable level of security without sacrificing too much the performances.
je: The jemalloc allocator by Jason Evans, now developed at Facebook and widely used in practice, for example in FreeBSD and Firefox.
lf: The lockfree-malloc allocator, multi-thread scalability-focused.
lp: The libpas allocator, used by WebKit.
lt: The ltalloc allocator, a multi-threaded memory allocator based on free lists best suited for many small allocations.
mesh: The mesh allocator, a memory allocator that automatically reduces the memory footprint of C/C++ applications. Also tested as nomesh with the meshing feature disabled.
mi: The mimalloc allocator. We can also test the debug version as dmi (this can be used to check for any bugs in the benchmarks), and the secure version as smi.
mng: musl's memory allocator.
pa: The PartitionAlloc allocator used in Chromium.
rp: The rpmalloc allocator uses 16-byte aligned allocations and is developed by Mattias Jansson at Epic Games, used for example in Haiku.
sc: The scalloc allocator, a fast, multicore-scalable, low-fragmentation memory allocator
scudo: The scudo allocator used by Fuschia and Android.
sg: The slimguard allocator, designed to be secure and memory-efficient.
sm: The Supermalloc allocator by Bradley Kuszmaul uses hardware transactional memory to speed up parallel operations.
sn: The snmalloc allocator is a recent concurrent message passing allocator by Liétar et al. [8].
tbb: The Intel TBB allocator that comes with the Thread Building Blocks (TBB) library [7].
tc: The tcmalloc allocator which comes as part of the Google performance tools, now maintained by the commuity.
tcg: The tcmalloc allocator, maintained and used by Google.
yal: The yalloc yet another allocator aims at balancing safety and compactness.
sys: The system allocator. Here we usually use the glibc allocator (which is originally based on Ptmalloc2).

Current Benchmarks

The first set of benchmarks are real world programs, or are trying to mimic some, and consists of:

barnes: a hierarchical n-body particle solver [4], simulating the gravitational forces between 163840 particles. It uses relatively few allocations compared to cfrac and espresso but is multithreaded.
cfrac: by Dave Barrett, implementation of continued fraction factorization, using many small short-lived allocations.
espresso: a programmable logic array analyzer, described by Grunwald, Zorn, and Henderson [3]. in the context of cache aware memory allocation.
gs: have ghostscript process the entire Intel Software Developer’s Manual PDF, which is around 5000 pages.
leanN: The Lean compiler by de Moura et al, version 3.4.1, compiling its own standard library concurrently using N threads (./lean --make -j N). Big real-world workload with intensive allocations.
redis: running redis-benchmark, with 1 million requests pushing 10 new list elements and then requesting the head 10 elements, and measures the requests handled per second. Simulates a real-world workload.
larsonN: by Larson and Krishnan [2]. Simulates a server workload using 100 separate threads which each allocate and free many objects but leave some objects to be freed by other threads. Larson and Krishnan observe this behavior (which they call bleeding) in actual server applications, and the benchmark simulates this.
larsonN-sized: same as the larsonN except it uses sized deallocation calls which have a fast path in some allocators.
lua: compiling the lua interpreter.
z3: perform some computations in z3.

The second set of benchmarks are stress tests and consist of:

alloc-test: a modern allocator test developed by OLogN Technologies AG (ITHare.com) Simulates intensive allocation workloads with a Pareto size distribution. The alloc-testN benchmark runs on N cores doing 100·10⁶ allocations per thread with objects up to 1KiB in size. Using commit 94f6cb (master, 2018-07-04)
cache-scratch: by Emery Berger [1]. Introduced with the Hoard allocator to test for passive-false sharing of cache lines: first some small objects are allocated and given to each thread; the threads free that object and allocate immediately another one, and access that repeatedly. If an allocator allocates objects from different threads close to each other this will lead to cache-line contention.
cache_trash: part of Hoard benchmarking suite, designed to exercise heap cache locality.
glibc-simple and glibc-thread: benchmarks for the glibc.
kq_bench_08/09/12: the malloc_benchmark benchmark from boostkit/kqmalloc v0.19.0-beta0, built as extern/kq_bench with make clean kunpeng. These group aliases expand to kq_bench_<hip>_m0, kq_bench_<hip>_m1, and kq_bench_<hip>_m2; each item runs one mode with -threads N -affinity 0.
malloc-large: part of mimalloc benchmarking suite, designed to exercice large (several MiB) allocations.
mleak: check that terminate threads don't "leak" memory.
rptest: modified version of the rpmalloc-benchmark suite.
mstress: simulates real-world server-like allocation patterns, using N threads with with allocations in powers of 2 where objects can migrate between threads and some have long life times. Not all threads have equal workloads and after each phase all threads are destroyed and new threads created where some objects survive between phases.
rbstress: modified version of allocator_bench, allocates chunks in memory via ruby shenanigans.
sh6bench: by MicroQuill as part of SmartHeap. Stress test where some of the objects are freed in a usual last-allocated, first-freed (LIFO) order, but others are freed in reverse order. Using the public source (retrieved 2019-01-02)
sh8benchN: by MicroQuill as part of SmartHeap. Stress test for multi-threaded allocation (with N threads) where, just as in larson, some objects are freed by other threads, and some objects freed in reverse (as in sh6bench). Using the public source (retrieved 2019-01-02)
xmalloc-testN: by Lever and Boreham [5] and Christian Eder. We use the updated version from the SuperMalloc repository. This is a more extreme version of the larson benchmark with 100 purely allocating threads, and 100 purely deallocating threads with objects of various sizes migrating between them. This asymmetric producer/consumer pattern is usually difficult to handle by allocators with thread-local caches.
recommend-like: jemalloc-specific KPI bench simulating search/recommendation request workloads — short-lived objects, high churn, steady-state dirty pages. Reads 5 jemalloc stats via mallctl (Allocated/RSS, Dirty/RSS, Metadata/Allocated, ChurnRate, 16-32K alloc count) and validates whether the bench shape matches real-service profiles. Three entry points: run_sanity.sh (single-run KPI PASS/FAIL), run.sh (B0 vs A multi-round compare with MAD/median noise floor), and the main bench.sh recommend-like driver — see bench/recommend-like/README.md. Jemalloc-only: the bench binary links jemalloc at compile time for the stats interface, so bench.sh filters alloc_run against a jemalloc-family whitelist (je myje myje-base, overridable via RECOMMEND_LIKE_ALLOCS) and warns + skips on other allocators to avoid writing meaningless KPI rows into benchres.csv. Example: ../../bench.sh je recommend-like (OK), ../../bench.sh tc mi recommend-like (skipped).

Finally, there is a security benchmark aiming at checking basic security properties of allocators.

Benchmark Evaluation Dimensions

The full benchmark suite covers the following key evaluation dimensions:

Dimension	Benchmarks
Single-thread throughput	cfrac, espresso, glibc-simple, alloc-test1
Multi-thread scalability	leanN, alloc-testN, sh8bench, xmalloc-testN, glibc-thread
Cross-thread object migration	larson, sh8bench, xmalloc-test, mstress
Cache locality / false sharing	cache-scratch, cache-thrash, espresso
Large allocation efficiency	malloc-large
Memory footprint / fragmentation	all benchmarks (RSS peak recorded)
Thread-exit memory reclamation	mleak
Real-world application workloads	redis, gs, lean, lua, linux, rocksdb, z3, spec
Security properties	security (double-free, overflow, underflow, use-after-free, …)

Benchmark Ranking: Generality and Discriminative Power

The table below ranks each benchmark by two criteria:

Generality — how broadly representative the workload is of real-world usage
Effectiveness — how well the benchmark differentiates allocator quality

Rank	Benchmark	Generality	Effectiveness	Rationale
1	leanN	★★★★★	★★★★★	Large real-world multi-threaded compiler workload with intensive allocations; best at exposing practical performance differences
2	alloc-testN	★★★★☆	★★★★★	Pareto-distributed sizes mimic realistic programs; 100M allocs/thread sharply differentiates throughput
3	larson / larson-sized	★★★★★	★★★★☆	Classic server cross-thread "bleeding" pattern; used in nearly every allocator paper
4	cfrac	★★★★☆	★★★★☆	Canonical single-thread small-object benchmark; clearly exposes single-threaded allocation efficiency
5	xmalloc-testN	★★★☆☆	★★★★★	Extreme producer/consumer asymmetry; most effective at exposing weaknesses in thread-local caching strategies
6	cache-scratch	★★★☆☆	★★★★★	Precisely measures passive false sharing; the only test that effectively detects this critical issue
7	redis	★★★★★	★★★☆☆	High real-world relevance (server workload), but allocator differences are typically small
8	mstress	★★★★☆	★★★★☆	Multi-phase thread recreation with object migration; good all-round stress test
9	espresso	★★★★☆	★★★☆☆	Classic cache-locality benchmark, but single-threaded nature limits discriminative power
10	rptest	★★★☆☆	★★★★☆	Highly configurable; covers multiple dimensions, but use-case is narrower
11	linux	★★★★★	★★★☆☆	Most realistic build workload; allocator impact is often overshadowed by I/O
12	rocksdb	★★★★☆	★★★☆☆	Good database-workload coverage; bottleneck is often I/O rather than allocation
13	sh6bench / sh8bench	★★★☆☆	★★★☆☆	Classic stress tests with synthetic free-order patterns
14	malloc-large	★★★☆☆	★★★☆☆	Specific but important large-allocation scenario
15	barnes	★★★☆☆	★★☆☆☆	Too few allocations to meaningfully differentiate allocators
16	security (135 tests)	★★★★☆	★★★★★	Unique safety dimension; measures security properties rather than performance
17	mleak	★★☆☆☆	★★★★☆	Narrow scenario, but the thread-exit leak issue it detects is critical
18	spec	★★★★★	★★★★☆	Industry-standard suite; requires a separate SPEC CPU 2017 installation
19	glibc-simple / glibc-thread	★★★☆☆	★★☆☆☆	Simple glibc-derived patterns with limited discriminative power
20	z3, gs, lua, lean-mathlib, rbstress, cache-thrash	★★☆☆☆	★★☆☆☆	Supplementary tests covering long-tail scenarios

Quick Benchmark Set (`quickt`)

For a fast but representative evaluation, use the quickt option and append the matching KQ benchmark group explicitly:

~/dev/mimalloc-bench/out/bench> ../../bench.sh --procs=`nproc` -r=5 -n=1 -s=1 kq_09 je tc mi quickt kq_bench_09

This runs carefully selected benchmarks that together cover the most important and distinct evaluation dimensions:

Benchmark	Key Dimension	Why Selected
cfrac	Single-thread small-object throughput	The canonical baseline for pure allocation speed; fast and low-noise
alloc-testN	Multi-thread scalability	Pareto size distribution; 100M allocs/thread yields the clearest throughput signal
larson-sized	Cross-thread object migration + sized deallocation	Represents server workloads where objects are freed by threads other than their allocator; also exercises the sized-free fast path
leanN	Real-world dense multi-thread allocation	Large-scale compiler workload; most predictive of real application performance
redis	Real-world single-threaded workload	High real-world relevance (server workload)
rocksdb	Real-world storage engine workload	Good database-workload coverage; bottleneck is often I/O rather than allocation; but particularly effective for validating memory usage characteristics
kq_bench_08/09/12	KQ malloc stress modes	Optional group aliases that run the kqmalloc `malloc_benchmark` modes using explicit HIP08, HIP09, or HIP12 binaries

One-shot runner: `run_quickt.sh`

# Default: auto-detects KQ allocator and kq_bench variants from CPU part, plus je tc mi
./run_quickt.sh

# Explicit allocator list
./run_quickt.sh kq_09 je tc mi

# Override rounds and core count
ROUNDS=3 PROCS=4 ./run_quickt.sh kq_09 je tc mi

The script auto-detects the KQ allocator and matching kq_bench variant from /proc/cpuinfo:

CPU part	Allocator	kq_bench
`0xd06`	`kq_12`	`kq_bench_12`
`0xd02`	`kq_09`	`kq_bench_09`
other	`kq_08`	`kq_bench_08`

After the run, scripts/analyze_bench.py prints median, MAD, and ratio tables to stdout and appends them to out/bench/benchres.csv.

Example

bench-z4-1 bench-z4-2

Memory usage:

Results and notable usages

Improvements

Minor performances improvement in isoalloc
Parallel compilation support in DieHarder
Portability improvement in Intel TBB malloc
Various portability improvements in Google's tcmalloc
Improved double-free detection in snmalloc
Fixed compilation on modern glibc in SlimGuard
A crash in isoalloc
Caught a compilation issue in rpmalloc
Portability issues in rpmalloc

Notable usages and papers

Provided data for the glib allocator.
Provided data for snmalloc hardening.
Used as main benchmark suite by S2malloc: Statistically Secure Allocator for Use-After-Free Protection And More by Ruizhe Wang, Meng Xu and N. Asokan
Used as benchmark suite by StarMalloc: Verifying a Modern, Hardened Memory Allocator by Antonin Reitz, Aymeric Fromherz, and Jonathan Protzenko

References

[1] Emery D. Berger, Kathryn S. McKinley, Robert D. Blumofe, and Paul R. Wilson. Hoard: A Scalable Memory Allocator for Multithreaded Applications the Ninth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS-IX). Cambridge, MA, November 2000. pdf
[2] P. Larson and M. Krishnan. Memory allocation for long-running server applications. In ISMM, Vancouver, B.C., Canada, 1998. pdf
[3] D. Grunwald, B. Zorn, and R. Henderson. Improving the cache locality of memory allocation. In R. Cartwright, editor, Proceedings of the Conference on Programming Language Design and Implementation, pages 177–186, New York, NY, USA, June 1993. pdf
[4] J. Barnes and P. Hut. A hierarchical O(n*log(n)) force-calculation algorithm. Nature, 324:446-449, 1986.
[5] C. Lever, and D. Boreham. Malloc() Performance in a Multithreaded Linux Environment. In USENIX Annual Technical Conference, Freenix Session. San Diego, CA. Jun. 2000. Available at https://github.com/kuszmaul/SuperMalloc/tree/master/tests
[6] Timothy Crundal. Reducing Active-False Sharing in TCMalloc. 2016. http://courses.cecs.anu.edu.au/courses/CSPROJECTS/16S1/Reports/Timothy*Crundal*Report.pdf. CS16S1 project at the Australian National University.
[7] Alexey Kukanov, and Michael J Voss. The Foundations for Scalable Multi-Core Software in Intel Threading Building Blocks. Intel Technology Journal 11 (4). 2007
[8] Paul Liétar, Theodore Butler, Sylvan Clebsch, Sophia Drossopoulou, Juliana Franco, Matthew J Parkinson, Alex Shamis, Christoph M Wintersteiger, and David Chisnall. Snmalloc: A Message Passing Allocator. In Proceedings of the 2019 ACM SIGPLAN International Symposium on Memory Management, 122–135. ACM. 2019.

Similar repos

daanx/mimalloc-bench

Suite for benchmarking malloc implementations.

516

microsoft/mimalloc

mimalloc is a compact general purpose allocator with excellent performance.

13,392

emeryberger/Hoard

The Hoard Memory Allocator: A Fast, Scalable, and Memory-efficient Malloc for Linux, Windows, and…

1,250

mjansson/rpmalloc-benchmark

Benchmarks for our memory allocator

mjansson/rpmalloc

Public domain cross platform lock free thread caching 16-byte aligned memory allocator implemented…

2,508

jemalloc/jemalloc

11,190

bloomberg/bde-allocator-benchmarks

A set of benchmarking tools used to quantify the performance of BDE-style polymorphic allocators.

evanwashere/mitata

benchmark tooling that loves you ❤️

2,534

Contributors

215 commits

169 commits

25 commits

13 commits

Languages

81.4%

Shell

7.4%

C++

7.0%

Python

3.4%

kunpengcompute/malloc-bench

README

Mimalloc-bench

Benchmarking

Current Allocators

Current Benchmarks

Benchmark Evaluation Dimensions

Benchmark Ranking: Generality and Discriminative Power

Quick Benchmark Set (quickt)

One-shot runner: run_quickt.sh

Example

Results and notable usages

Improvements

Notable usages and papers

References

Contributors

Similar repos

Languages

kunpengcompute/malloc-bench

README

Mimalloc-bench

Benchmarking

Current Allocators

Current Benchmarks

Benchmark Evaluation Dimensions

Benchmark Ranking: Generality and Discriminative Power

Quick Benchmark Set (quickt)

One-shot runner: run_quickt.sh

Example

Results and notable usages

Improvements

Notable usages and papers

References

Similar repos

Contributors

Languages

Quick Benchmark Set (`quickt`)

One-shot runner: `run_quickt.sh`

Quick Benchmark Set (`quickt`)

One-shot runner: `run_quickt.sh`