A collection of benchmarking tools and libraries for testing,
comparing, and improving the performance of SurrealDB.

Benchmarks

This repository exists as a central repository for a collection of benchmarking tools and libraries for testing, comparing, and improving the performance of SurrealDB. Each tool linked to from this location exists within its own repository.

As we add support for additional benchmarking suites, we will add and link to them in this repository.

For SurrealDB performance best practices, see our documentation.

crud-bench

The crud-bench benchmarking tool is an open-source benchmarking tool for testing and comparing the performance of a number of different workloads on embedded, networked, and remote databases. It can be used to compare both SQL and NoSQL platforms including key-value, and embedded databases. Importantly crud-bench focuses on testing additional features which are not present in other benchmarking tools, but which are available in SurrealDB.

The primary purpose of crud-bench is to continually test and monitor the performance of features and functionality built in to SurrealDB, enabling developers working on features in SurrealDB to assess the impact of their changes on database queries and performance.

The crud-bench benchmarking tool is being actively developed with new features and functionality being added regularly.

git clone -b main https://github.com/surrealdb/crud-bench

cargo build --release && ln -s target/release/crud-bench crud-bench

crud-bench --help

dockerd

crud-bench -d surrealdb-rocksdb -s 5000000 -c 128 -t 48 -k string26 -r

crud-bench -d surrealdb --endpoint ws://127.0.0.1:8000 -s 5000000 -c 128 -t 48 -k string26 -r

ann-benchmarks

The Approximate Nearest Neighbour benchmarks suite is a testing tool for comparing the performance of vector datasets, with a focus on approximate and exact nearest neighbour algorithms across different data platforms, databases, and libraries. This project contains tools to benchmark various implementations of approximate nearest neighbor, and exact nearest neighbour search for selected metrics - with pre-generated datasets, and a test suite to verify function integrity.

git clone -b surrealdb https://github.com/surrealdb/ann-benchmarks

python3.10 -m venv path/to/venv && source path/to/venv/bin/activate

pip3.10 install -r requirements.txt

python3.10 install.py --algorithm surreal_hnsw

python3.10 run.py --run-disabled --algorithm surreal_hnsw --dataset random-xs-20-euclidean

go-ycsb

The Yahoo! Cloud Serving Benchmark (YCSB) is an open-source specification and program suite for evaluating retrieval and maintenance capabilities of computer programs. It is often used to compare the relative performance of NoSQL database management systems. This Golang port of the original benchmarking suite allows benchmarking comparison using the Golang programing language, and is sponsored by Pingcap.

git clone -b surrealdb https://github.com/surrealdb/go-ycsb

make quick

./bin/go-ycsb load surrealdb -P workloads/workloada
./bin/go-ycsb run surrealdb -P workloads/workloada

./bin/go-ycsb load surrealdb -P workloads/workloada -p surrealdb.uri='ws://127.0.0.1:8000' -p surrealdb.user=root -p surrealdb.pass=root
./bin/go-ycsb run surrealdb -P workloads/workloada -p surrealdb.uri='ws://127.0.0.1:8000' -p surrealdb.user=root -p surrealdb.pass=root

nosqlbench

[!IMPORTANT] The SurrealDB changes to this benchmarking tool have not yet been released. Please check back later.

NoSQLBench is a serious performance testing tool for the NoSQL ecosystem, sponsored by DataStax. It brings together features and capabilities that are not found in any other tool. The core machinery of NoSQLBench has been built with attention to detail. It has been battle tested within DataStax and in the NoSQL ecosystem as a way to help users validate their data models, baseline system performance, and qualify system designs for scale.