clink

clink is an embedded-first, Arrow-native stream processing engine in modern C++ (C++23): stateful stream processing with engine-grade semantics - SQL, event time, keyed state, exactly-once checkpoints - that you run like a tool rather than operate like a platform.

The whole engine lives in one library, and the same pipeline runs two ways. In-process: clink run pipeline.sql executes it in a single process with no daemons and prints its first result about 155 ms after process start; libclink embeds the engine in any service behind a pure-C ABI with results as Arrow C streams; pyclink returns them as pyarrow tables in a notebook. At scale: the same SQL file, unchanged, submits to a distributed Coordinator/Worker cluster with parallelism, failover, and rescale.

Three capabilities follow from that design:

• State is an open dataset. Snapshots are documented Arrow IPC: checkpoints and savepoints open directly in pyarrow, DuckDB or Polars, export to Parquet and Iceberg, and a running job's live state serves point lookups and Arrow scans over plain HTTP, no sink round-trip. See state and backends.
• Incidents replay deterministically. A flight recorder captures what each operator consumed per checkpoint epoch; clink replay re-executes an operator over exactly those records, offline and byte-identically, and can freeze the incident into a permanent regression test. See replay determinism.
• Nothing to manage under it. A single static binary with no managed runtime: cold start in milliseconds, one artefact to ship, and the same behaviour embedded, in CI, and on a cluster.

Measured, not asserted: across the 17-query nexmark suite on a five-node cluster, clink processes an event for 1.9x to 5.3x less CPU (median 2.45x) than a JVM stream processor producing identical, correctness-gated output. Method, caveats and raw per-run data: Benchmarks, priced out in instances, dollars and modelled CO2e at Cost and environmental footprint.

clink is heavily inspired by Apache Flink. Flink's model of typed operator DAGs, event-time processing, in-band watermarks and checkpoint barriers, keyed state, and exactly-once semantics is the conceptual foundation this engine builds on. clink reworks that model in modern C++ around Arrow-native columnar execution and JVM-free deployment.

Try it in one command

Nothing to clone, install or configure:

docker run --rm ghcr.io/orhaugh/clink-runtime:latest run /opt/clink/examples/sql/hello.sql

The pipeline and the data it reads both ship inside the image (amd64 and arm64). It groups a small stream of device readings by region and prints the result:

{"avg_reading":21.4,"peak":21.4,"readings":1,"region":"emea"}
{"avg_reading":22.65,"peak":23.9,"readings":2,"region":"emea"}
{"avg_reading":19.2,"peak":19.2,"readings":1,"region":"amer"}
...

Those rows are the point. A batch engine prints one line per region at the end; clink prints a line per update, because the aggregate is a stream and you are watching it change. Run your own file the same way by mounting it:

docker run --rm -v "$PWD:/work" ghcr.io/orhaugh/clink-runtime:latest run /work/mine.sql

Want to build something real? Follow the Kafka to clink to ClickHouse tutorial: one docker compose up gives you a broker, a database and a small clink cluster running an event-time windowed pipeline; you then kill the Worker mid-stream, watch it recover from its checkpoint, verify every result independently, and open the job's state as an Arrow table. The example lives in examples/kafka-to-clickhouse.

Build from source

clink builds against a pinned Apache Arrow toolchain in ~/.clink-deps. The bootstrap step downloads a prebuilt, checksum-pinned archive when one exists for your platform (macOS arm64, Linux x86_64/arm64; about a minute) and compiles it from source otherwise:

git clone https://github.com/orhaugh/clink && cd clink
scripts/build-arrow.sh && scripts/build-iceberg-cpp.sh   # one-time, cached
cmake -S . -B build && cmake --build build --parallel 10
ctest --test-dir build --parallel 8                      # optional

Then run a first pipeline - one process, no daemons:

printf '{"usr":"alice","amount":12}\n{"usr":"bob","amount":7}\n{"usr":"alice","amount":5}\n' > /tmp/orders.ndjson

./build/clink run -e "CREATE TABLE orders (usr VARCHAR, amount BIGINT) \
      WITH (connector='file', format='json', path='/tmp/orders.ndjson'); \
    SELECT usr, SUM(amount) AS total FROM orders GROUP BY usr"

Supported platforms: macOS (Apple Silicon is the primary development platform) and Linux (Debian-family, exercised in CI). Windows is not supported. ./build_and_test.sh is the reproducible CI-matching path, including the sanitizer matrix; CONTRIBUTING.md has the details.

One engine, two execution models

Embedded. clink run pipeline.sql runs the whole engine in one process: SQL frontend, operators, state backends, checkpointing, connectors. First result in about 155 ms from process start, a figure gated by a release test so it cannot silently regress. The same engine embeds behind a pure-C ABI (libclink), from Python (pyclink), and over Arrow Flight SQL for any ADBC/JDBC client.

Distributed. The same SQL file or compiled job plugin submits to a Coordinator/Worker cluster: parallel subtasks, hash-partitioned keyed shuffles, exactly-once checkpoints, failover from the last completed checkpoint, hot per-operator rescale, and rolling upgrades through savepoints. A Helm chart and Kubernetes operator ship in-tree.

clink_node --role=coordinator --rpc-port=6123 &
clink_node --role=worker --coordinator-host=127.0.0.1 --coordinator-port=6123 &
clink run pipeline.sql --coordinator-host=127.0.0.1 --coordinator-port=8081

Typed C++ pipelines (the fluent Pipeline / DataStream<T> API, keyed process functions, windows, joins, CEP) are documented with runnable programs under docs/consumer-examples/.

Production qualification

Benchmarks say how fast an engine is; they say nothing about whether its guarantees hold when processes die at the worst possible instant. clink runs a standing qualification programme: long campaigns on disposable multi-host rigs with faults injected into the narrowest windows of the engine's own protocols, judged by an independent oracle, published only when green and with the evidence retained. A few of the published results:

The full table, the rig, the method and every campaign's honesty-bounded claims: Qualification.

Status and maturity

clink is young and pre-1.0. Its guarantees are qualified within the published bounds above, not battle-tested by years of third-party production deployments; public C++ APIs may still change between minor releases, and every such change is called out in the CHANGELOG. Durable state is treated conservatively: snapshots carry schema versions with a migrate-at-restore path, so an upgrade does not silently invalidate checkpoints or savepoints. What the engine can do, feature by feature with caveats stated, lives in the capability catalogue.

Road to 1.0

The path to 1.0 is primarily about stability and operational evidence rather than adding another broad layer of features.

Before 1.0, the priorities are:

• Long-duration qualification. Complete multi-day production-style campaigns and continue expanding the measured state, scale and recovery envelope.
• Stable public APIs. Settle the C++, C and SQL-facing interfaces that should carry compatibility guarantees across the 1.x line.
• Stable extension model. Reduce coupling between compiled plugins and individual clink revisions, with a clearly defined compatibility contract.
• Native type ergonomics. Make a single schema declaration sufficient for Arrow conversion, state serialisation and distributed type registration (design record 009).
• Operational maturity. Incorporate real-world deployment feedback, strengthen upgrade and diagnostic tooling, and close issues found by external workloads.

1.0 will mean that these compatibility and operational contracts are ready to be relied upon. It will not mean that feature development stops.

See GitHub issues for work currently in progress.

Documentation

Everything below is published at orhaugh.github.io/clink:

• Capability catalogue: the complete shipped feature surface, with caveats.
• SQL reference: the supported SQL surface, from DDL through windows, joins, MATCH_RECOGNIZE, UDFs and SQL-native ML.
• Connectors: twenty-plus sources and sinks (Kafka, Postgres incl. CDC, ClickHouse, S3/GCS/Azure Parquet, Iceberg, MQTT, NATS, Pulsar, RabbitMQ, Redis, MongoDB, Cassandra, HTTP, Avro, WebSocket and more), each with dependencies, options and delivery semantics.
• Internals: every subsystem documented the way its code is structured, citing sources.
• Design decisions: why the engine is built this way, trade-offs included.
• Benchmarks and cost footprint.
• Qualification.
• Tutorial: Kafka to clink to ClickHouse on your machine, with a deliberate Worker kill and independent verification (examples/kafka-to-clickhouse).
• Runnable examples: every core feature as a standalone buildable program, from hello-pipeline to the testing framework and state-as-data workflows.

Installing

clink ships as a regular CMake package:

cmake -S . -B build -DCLINK_BUILD_TESTS=OFF -DCLINK_BUILD_EXAMPLES=OFF \
                    -DCMAKE_INSTALL_PREFIX=/usr/local
cmake --build build --parallel 10
sudo cmake --install build

Downstream projects consume it with find_package(clink REQUIRED) and link clink::clink (or per-impl targets such as clink::kafka); docs/consumer-examples/ is a complete find_package-based project to copy from. The install carries the clink CLI and the clink_node cluster daemon. Optional connectors and backends are CLINK_WITH_<NAME> CMake options (default AUTO: used when the dependency is found); the SQL frontend is CLINK_BUILD_SQL=ON by default. clink --capabilities prints what any given binary was built with.

Getting help

• Bug reports: GitHub issues.
• Questions and ideas: GitHub discussions.
• Contributing a change: CONTRIBUTING.md.
• Reporting a security issue privately: SECURITY.md.

Licence and attribution

clink is licensed under the Apache License 2.0. See LICENSE and NOTICE. You are free to use clink for any purpose, including in commercial and closed-source products. If you build on it, a link back to this repository is appreciated; for write-ups, please cite it via CITATION.cff.