vesiro-benchmarker

vesiro-benchmarker is a command-line benchmarking tool for Elasticsearch.

Build

Building requires Go 1.23 or newer. Run these commands from the repository root:

make build
./bin/bench --help

This builds the program at bin/bench. You can also build without make using go build -o bin/bench ./cmd/bench.

Run your first benchmark

You'll need a running Elasticsearch node with data already in an index.

Start with the included match_all.json. It works with any index because it doesn't depend on specific fields. It matches all documents and returns a page of hit details.

Send one request to check the connection. Replace my-index with your index name and http://localhost:9200 with your node's address:

./bin/bench single \
  --node-url=http://localhost:9200 \
  --index-name=my-index \
  --query-template=assets/query/templates/match_all.json

--query-template is the path to a query template file. single prints the query, HTTP status, response time, and response body. If your node needs a login, add --user='username:password' and use an https:// URL for HTTPS, or keep them in a config file (see Saved defaults).

Once that works, send 100 requests from each of four clients:

./bin/bench run \
  --node-url=http://localhost:9200 \
  --index-name=my-index \
  --query-template=assets/query/templates/match_all.json \
  --num-clients=4 \
  --requests-per-client=100

A client sends one request, waits for the response, then sends the next. Four clients means up to four requests can be in progress at once, all from a single bench process. This run sends 400 search requests in total.

Read the results

Here's an example report for a run with 400 requests:

Requests:        400
Ok:              400
Errors:          0 (0.00%)
Statuses:        200: 400
q/s:             250.00
Execution Time:  1.6s
Took Avg:        12.00ms
Client Latency:  avg 15.00ms  p50 14.00ms  p95 24.00ms  p99 30.00ms
HasHits:         400 (100.0%)
Per Query (client latency, ms):
  Query           Requests    Avg    p50    p95    p99
  match_all.json       400  15.00  14.00  24.00  30.00

Client latency includes the network trip. The node's took covers its own work, so the two measure different things.

Check errors before comparing timings. Failed search responses are included in the timings. Connection failures, request timeouts, or invalid response JSON stop the run; those requests aren't counted in the report.

Every run also saves its report as JSON, so you can compare it with other runs later. See Save and compare runs.

Common Crawl queries

The cc-wet folder contains 80 query templates for Common Crawl WET data, grouped by query type.

These templates need an index with the mappings defined in cc-wet.json. You can use vesiro-indexer to set up an index with Common Crawl data.

Change the load

Set a request count, a run duration, or both. With both set, the run stops at whichever limit it reaches first. You can also stop a run with Ctrl-C and get a report of the results collected so far.

Without --qps, each client sends its next request as soon as the previous one finishes. To limit the average rate to 200 requests per second for one minute, run:

./bin/bench run \
  --node-url=http://localhost:9200 \
  --index-name=my-index \
  --query-template=assets/query/templates/match_all.json \
  --num-clients=16 \
  --benchmark-timeout=60 \
  --qps=200

Choose your queries

The four commands use the same query template files:

Define a query template

A template lets you change parts of the query, such as the search terms. The query goes inside a template field, and bindings supplies the values to fill in before each request.

For example, match_or.json searches the content field with phrases from a text file. Its main fields are:

{
  "name": "match_or",
  "bindings": {
    "Phrase": {
      "source": "file",
      "path": "./assets/query/terms/phrase-2.txt"
    }
  },
  "template": {
    "_source": false,
    "query": {
      "match": { "content": { "query": "{{.Phrase}}" } }
    }
  }
}

File paths inside templates are relative to the directory where you run bench. Run the bundled examples from the repository root so their term files can be found.

Run a folder of queries

folder benchmarks every query file in a folder and in all of its subfolders. It runs the files one after another: all clients send the first query file until it reaches its limit, then move on to the next one. That way each query's results aren't affected by the others. You can use your own folder or one of the included query folders.

The request count or --benchmark-timeout applies to each query file, not to the whole run. This example sends 1,600 requests for each of the files in the folder:

./bin/bench folder \
  --node-url=http://localhost:9200 \
  --index-name=my-index \
  --query-folder=assets/query/templates/cc-wet/boolean \
  --num-clients=16 \
  --requests-per-client=100

To run all 80 Common Crawl templates, point --query-folder at the parent folder, assets/query/templates/cc-wet. With --benchmark-timeout=30 instead of a request count, each template runs for 30 seconds, so the whole run takes about 40 minutes.

Add --warmup=10 to warm up each query file for 10 seconds immediately before measuring it.

Run a mix of queries

Add --random to mix the query files instead. Each request then picks a query file at random, and the request count or --benchmark-timeout applies to the whole run. This example sends 1,600 requests in total, spread across the files in the folder:

./bin/bench folder \
  --node-url=http://localhost:9200 \
  --index-name=my-index \
  --query-folder=assets/query/templates/cc-wet/boolean \
  --num-clients=16 \
  --requests-per-client=100 \
  --random

The report shows how many times each query ran.

Save and compare runs

Every run and folder benchmark saves its report as a JSON file in the results folder. The file is named by when the run started, the command, and the index:

results/2026-09-24T14-15-02_folder_cc-wet_after-upgrade.json

A run stopped with Ctrl-C, or one that fails partway through, is still saved and marked as incomplete. To stop saving by default, set "save": false in config.json. You can then add --save to save a single run.

To compare two runs, pass the run to compare against as --baseline and the run to check as --contender:

./bin/bench compare \
  --baseline=results/2026-09-24T11-30-12_folder_cc-wet_before-upgrade.json \
  --contender=results/2026-09-24T14-15-02_folder_cc-wet_after-upgrade.json

Saved defaults

For settings you use often, copy config.example.json to config.json. Anything set there no longer needs to be passed as a command-line flag.

Best practises

• Keep warm-up in mind when comparing results. Early requests can be slower while the node's caches and runtime warm up.
• Keep the data, queries, client count, and background traffic consistent.
• Run bench on a separate machine when possible, so it doesn't compete with the search node for CPU and memory.

License

Apache License 2.0.