DortDB

A modular, multi-language query engine for the JavaScript data already in your app's memory.

📖 Documentation  ·  🕹️ Live demo  ·  🎓 Thesis

DortDB queries arrays, DOM trees, and graphs in place, without moving them into a separate database process. SQL, Cypher, and XQuery come with it, you can mix them in a single query, and you can add a language of your own. Every language compiles to one shared algebra, so the optimizer plans and the executor runs a cross-model query as a single plan.

import { DortDB } from '@dortdb/core';
import { defaultRules } from '@dortdb/core/optimizer';
import { SQL } from '@dortdb/lang-sql';

const db = new DortDB({ mainLang: SQL(), optimizer: { rules: defaultRules } });

db.registerSource(['users'], [
  { name: 'Alice', age: 30 },
  { name: 'Bob', age: 25 },
]);

db.query('SELECT name FROM users WHERE age > 27');
// -> { schema: ['name'], data: [{ name: 'Alice' }] }

Contents

• Is DortDB for you?
• Installation
• Multiple languages in one query
• Data adapters
• Secondary indices
• Extensions
• Packages
• Documentation
• Alternatives

Is DortDB for you?

DortDB lets you:

• Query data that is already in memory, such as arrays, DOM and XML trees, and graphs, with no import or copy step.
• Mix relational, document, and graph queries in one query, each part in the language that suits it.
• Ship a small, tree-shakeable engine to the browser or Node, bundling only the languages you use.
• Extend the engine with your own languages, functions, indices, or optimizer rules.

DortDB does not:

• Provide persistence, transactions, or data modification. It reads in-memory data and nothing else.
• Optimize by cost, or spread execution over threads. The optimizer is rule-based and execution is single-threaded, which shows on very large datasets.
• Ship a large built-in library for each language yet. Add the functions you miss as an extension.

See Alternatives for libraries that may suit your use case better.

Installation

Install the core plus whichever language packages you need:

npm i @dortdb/core @dortdb/lang-sql
# add more languages when you need them
npm i @dortdb/lang-cypher graphology @dortdb/lang-xquery

Only the Cypher package needs Graphology, as a peer dependency.

Multiple languages in one query

The feature that sets DortDB apart is embedding one language inside another. A LANG block switches languages, and the inner query can read values from the surrounding scope. Everything lowers to the same algebra, so the engine optimizes and runs the whole query as one plan instead of as opaque nested calls.

const db = new DortDB({
  mainLang: SQL(),
  additionalLangs: [XQuery()],
  optimizer: { rules: defaultRules },
});

db.registerSource(['users'], [/* ... */]);
db.registerSource(['invoices'], new DOMParser().parseFromString('...', 'text/xml'));

// SQL filters users by a count computed with XQuery over an XML document
db.query(`
  SELECT name, age
  FROM users
  WHERE age > 30 AND (
    LANG xquery
    fn:count($invoices/customer[. = $users:name])
  ) > 5
`);

• A block starts with LANG <name> and ends at its enclosing scope (such as a closing parenthesis) or an explicit LANG EXIT.
• Blocks can nest to any depth, and inner queries can read values from any outer scope.

See Cross-language Queries for the full syntax and scope rules.

Data adapters

A data adapter separates a language from the concrete shape of your data, so you can point a language at sources of another shape. This one teaches SQL to read Map-backed rows instead of plain objects.

const db = new DortDB({
  mainLang: SQL({
    adapter: {
      createColumnAccessor: (prop) => (row: Map<string, unknown>) => row.get(prop),
      createRow: (keys, values) => new Map(keys.map((k, i) => [k, values[i]])),
    },
  }),
  optimizer: { rules: defaultRules },
});

db.registerSource(['users'], [
  new Map([['name', 'Alice'], ['age', 30]]),
  new Map([['name', 'Bob'], ['age', 25]]),
]);

db.query('SELECT name, age FROM users WHERE age > 27');
// -> { schema: ['name', 'age'], data: [{ name: 'Alice', age: 30 }] }

An adapter needs both members. createColumnAccessor reads a column out of a row, and createRow builds a row of the same shape when the engine constructs one itself. See Data Adapters.

Secondary indices

Register secondary indices for faster lookups and joins. The optimizer uses a matching index automatically.

import { DortDB, MapIndex } from '@dortdb/core';

db.registerSource(['users'], [/* ... */]);

// index a column, or any subquery-free expression
db.createIndex(['users'], ['age'], MapIndex);
db.createIndex(['users'], ['name[0] + age'], MapIndex);

An index class can also speed up joins over streams that have no index of their own. List those classes in hashJoinIndices to make them available to the executor.

new DortDB({
  mainLang: SQL(),
  optimizer: { rules: defaultRules },
  executor: { hashJoinIndices: [MapIndex] },
});

See Indexing & Performance.

Extensions

Package your own functions, operators, aggregates, and casts as an extension. The @dortdb/datetime extension in this repo adds date and time functions.

import { datetime } from '@dortdb/datetime';

const db = new DortDB({
  mainLang: SQL(),
  optimizer: { rules: defaultRules },
  extensions: [datetime],
});

db.query(`SELECT date.sub(now(), interval('3 years')) AS cutoff`);

See Extending DortDB.

Packages

The @dortdb/lang-cypher package is a set of implementation extensions to Cypher, based on the openCypher grammar, which is under the Apache License 2.0. The openCypher Implementers Group has not approved it. Cypher® is a registered trademark of Neo4j, Inc.

Documentation

The full documentation, the guides, and a generated API reference are at filipjezek.github.io/dortdb.

• Tutorials: Installation → Your First Query → Mixing Languages
• Guides: Running Queries, Cross-language Queries, Data Sources & Adapters, Indexing & Performance, Inspecting Plans
• Architecture and the Formalism
• Extending DortDB
• API reference

The thesis covers the design and the formal background in depth.

Alternatives

If DortDB is not the right tool for your use case, try one of these libraries.

SQL

• AlaSQL: A full SQL engine for Node and the browser, with persistence and data modification. It queries existing arrays too. It is larger and mostly slower than DortDB, and it has more features and built-in functions.
• sql.js: SQLite compiled to WebAssembly. A full, very fast SQL engine with persistence and data modification, but you have to copy your data into its database format.
• PGlite: PostgreSQL compiled to WebAssembly, with the same trade-off as sql.js.
• DuckDB-WASM: An analytical SQL engine compiled to WebAssembly. Many extensions and features, and again a copy into its own database format.

XQuery

• fontoxpath: An XQuery 3.1 engine for Node and the browser that can also modify data. Probably faster than DortDB, though I have not measured it.
• document.evaluate: The XPath engine built into browsers. It evaluates XPath only, not full XQuery.

No query language

• PouchDB: A lightweight database inspired by Apache CouchDB. It stores data in IndexedDB or WebSQL in the browser and syncs with a CouchDB server.
• RxDB: A reactive, offline-first database that syncs with a server. More features than PouchDB, and larger and more complex for it.

License

Released under the ISC License, except for the openCypher-derived grammar in @dortdb/lang-cypher, which is under the Apache License 2.0 (see that package's NOTICE).