Columnar analytics and graph traversal in one fused pipeline.

---

Rayforce is a pure C zero-dependency embeddable engine where columnar analytics and graph traversals share a single operation DAG, pass through a multi-pass optimizer, and execute as morsel-driven bytecode compiled at execution time. No malloc.

Install

Homebrew (macOS & Linux):

brew install rayforcedb/tap/rayforce

Debian / Ubuntu (.deb, x86-64) — grab the .deb from the latest release:

curl -LO https://github.com/RayforceDB/rayforce/releases/download/vX.Y.Z/rayforce_X.Y.Z_amd64.deb
sudo dpkg -i rayforce_X.Y.Z_amd64.deb

Prebuilt tarball (Linux x86-64 / macOS arm64) is also attached to each release. Or build from source below — zero dependencies, just make.

Quick Start

make            # debug build (ASan + UBSan)
make release    # optimized build
make test       # run full test suite
./rayforce      # start the Rayfall REPL

Rayfall REPL

Rayforce ships with Rayfall — a Lisp-like query language with a rich set of builtins. The REPL prompt is ‣:

‣ (set t (table [Symbol Side Qty]
    (list [AAPL GOOG MSFT AAPL GOOG]
          [Buy Sell Buy Sell Buy]
          [100 200 150 300 250])))

‣ (select {from:t by: Symbol Qty: (sum Qty)})
+--------+----------------------------+
| Symbol |            Qty             |
|  SYM   |            I64             |
+--------+----------------------------+
| AAPL   | 400                        |
| GOOG   | 450                        |
| MSFT   | 150                        |
+-------------------------------------+
| 3 rows (3 shown) 2 columns (2 shown)|
+-------------------------------------+

‣ (pivot t 'Symbol 'Side 'Qty sum)
+--------+-----+----------------------+
| Symbol | Buy |         Sell         |
|  SYM   | I64 |         I64          |
+--------+-----+----------------------+
| AAPL   | 100 | 300                  |
| GOOG   | 250 | 200                  |
| MSFT   | 150 | 0Nl                  |
+-------------------------------------+
| 3 rows (3 shown) 3 columns (3 shown)|
+-------------------------------------+

C API

Headers: include/rayforce.h (types, memory, atoms, vectors, tables, symbols), src/ops/ops.h (DAG construction, opcodes, optimizer, executor, graph algorithms), src/mem/heap.h (allocator lifecycle).

#include <rayforce.h>
#include "mem/heap.h"
#include "ops/ops.h"

int main(void) {
    ray_heap_init();
    ray_sym_init();

    /* Build a table */
    int64_t regions[] = {0, 0, 1, 1, 2, 2};
    int64_t amounts[] = {100, 200, 150, 300, 175, 225};
    ray_t* reg = ray_vec_from_raw(RAY_I64, regions, 6);
    ray_t* amt = ray_vec_from_raw(RAY_I64, amounts, 6);
    ray_t* tbl = ray_table_new(2);
    tbl = ray_table_add_col(tbl, ray_sym_intern("region", 6), reg);
    tbl = ray_table_add_col(tbl, ray_sym_intern("amount", 6), amt);
    ray_release(reg); ray_release(amt);

    /* Group by region, sum amounts */
    ray_graph_t* g = ray_graph_new(tbl);
    ray_op_t* keys[]    = { ray_scan(g, "region") };
    uint16_t  agg_ops[] = { OP_SUM };
    ray_op_t* agg_ins[] = { ray_scan(g, "amount") };
    ray_op_t* grp = ray_group(g, keys, 1, agg_ops, agg_ins, 1);

    ray_t* result = ray_execute(g, grp);

    if (result && !RAY_IS_ERR(result)) ray_release(result);
    ray_graph_free(g);
    ray_release(tbl);
    ray_sym_destroy();
    ray_heap_destroy();
}

How It Works

Build — Construct a lazy DAG: scans, filters, joins, aggregations, window functions, graph traversals. Nothing executes yet.

Optimize — Multi-pass rewriting: type inference → constant folding → idiom rewrite → SIP → factorize → predicate pushdown → filter reorder → projection pushdown → partition pruning → DCE.

Execute — Element-wise subtrees are compiled to flat bytecode at execution time and evaluated over 1024-element morsels that stay L1-resident, with no intermediate heap allocation. Radix-partitioned hash joins size partitions to fit L2. Thread pool dispatches morsels in parallel.

Features

Execution engine

• Lazy operation DAG — nothing runs until ray_execute
• Multi-pass optimizer with sideways information passing
• Expression compilation at execution time — element-wise subtrees compiled to flat bytecode, evaluated in 1024-row morsel batches with no intermediate materialization
• Radix-partitioned hash joins sized for L2 cache
• Thread pool with parallel morsel dispatch

Graph engine

• Double-indexed CSR storage (forward + reverse), mmap support
• BFS, DFS, Dijkstra, A*, PageRank, Louvain, Betweenness, LFTJ, and more
• Factorized execution avoids materializing cross-products
• SIP propagates selection bitmaps backward through expand chains

Rayfall language

• Arithmetic, string, aggregation, joins, higher-order, I/O builtins
• Lambdas compile lazily to bytecode, run in computed-goto VM
• select/update/pivot bridge to the DAG optimizer at runtime

Memory

• Buddy allocator with slab cache — O(1) for ~90% of allocations
• Thread-local arenas, lock-free allocation, COW ref counting
• No system allocator — ray_alloc/ray_free for everything

Vector search

• Multi-metric HNSW index (cosine / L2 / inner-product) with save/load
• Rayfall builtins: cos-dist / l2-dist / inner-prod / norm / knn and the HNSW lifecycle hnsw-build / ann / hnsw-save / hnsw-load / hnsw-free / hnsw-info
• Filter-aware ANN via select ... where ... nearest (ann handle query) take k
• Iterative streaming scan: the where predicate is pushed into HNSW's beam loop so rejected candidates don't consume result slots

Storage

• Columnar files with mmap, splayed tables, date-partitioned tables
• CSV reader with parallel mmap parse, type inference, null handling

Project Structure

include/rayforce.h         Single public header
src/mem/                    Buddy allocator, slab cache, arena, COW
src/core/                   Type system, platform abstraction, runtime
src/vec/                    Vector, list, string, selection bitmap ops
src/table/                  Table, symbol intern table
src/store/                  Column files, CSR, splayed/parted tables, HNSW
src/ops/                    DAG, optimizer, fused executor, LFTJ
src/io/                     CSV reader/writer (parallel mmap)
src/lang/                   Rayfall parser, evaluator, bytecode VM
src/app/                    REPL, terminal, pretty-printer
test/                       Test suites
examples/rfl/               Rayfall example scripts
examples/                   C API examples
website/                    Documentation site (GitHub Pages)

Documentation

Full docs: rayforcedb.github.io/rayforce

• Quick Start — build, REPL, first query
• Rayfall Language — syntax and builtins
• Data Types — types and collections
• Queries — select, joins, pivot, window
• C API — full API reference
• Graph Engine — algorithms
• Architecture — DAG, optimizer, memory

Python bindings

Rayforce has Python bindings at rayforce-py — contributions welcome.

Contributing

Contributions are welcome. You can help by:

• Reporting bugs and requesting features via GitHub Issues
• Submitting pull requests
• Creating example scripts and use cases
• Improving documentation

Development Partnership

Rayforce is jointly developed with and sponsored by Lynx.

This partnership has been instrumental in making Rayforce a mature, production-ready engine. Lynx's active involvement in development and their commitment to innovative open-source technologies in the financial sector has enabled Rayforce to reach its full potential.

License

MIT