duckdb-otlp

Your telemetry, in your own storage, answerable in SQL.

Receives OpenTelemetry over OTLP and OTAP and streams it into Parquet files, DuckLake, or Iceberg catalogs like Amazon S3 Tables and Cloudflare R2 Data Catalog — no collector to run, no cluster to size, no vendor to pay. Query it with DuckDB, or anything else that reads Parquet.

It ships two ways. As a DuckDB extension, it adds read_otlp_* / read_otap_* table functions and an embedded HTTP/gRPC ingest server to any DuckDB session. As a single binary (duckdb-otlp), it runs that same server standalone or in Docker, and doubles as a CLI for one-off conversion, export, and queries. Both speak OTLP and the OpenTelemetry Arrow protocol (OTAP).

Quickstart: Read OpenTelemetry data

Install and load the extension in duckdb v1.5.5 or higher:

-- Run commands inside the duckdb shell, install via `curl https://install.duckdb.org | sh` or `brew install duckdb`
-- See "Install pre-release extension via GitHub" to use a nightly builds
INSTALL otlp FROM community;
LOAD otlp;

-- Install unsigned extenstion from GitHub
-- You must start duckdb with `-unsigned` to allow this
INSTALL otlp from 'https://smithclay.github.io/duckdb-otlp';
LOAD otlp;

Read OTLP protobuf/JSON data from public URLs, local files, or object storage buckets:

-- Install extension to support reading over HTTP(S)
INSTALL httpfs; LOAD httpfs;

-- Read logs exported from the OpenTelemetry Collector
SELECT time_unix_nano, service_name, severity_text, body FROM read_otlp_logs('https://github.com/smithclay/duckdb-otlp/raw/refs/heads/main/test/data/otlp_logs.pb');

-- Read traces exported from the OpenTelemetry Collector
SELECT trace_id, name, duration_time_unix_nano FROM read_otlp_traces('https://github.com/smithclay/duckdb-otlp/raw/refs/heads/main/test/data/otlp_traces.pb') ORDER BY duration_time_unix_nano DESC;

Read the columnar OpenTelemetry Arrow Protocol (OTAP) with the read_otap_* readers. They emit the same schemas as read_otlp_*; pick the reader that matches your input encoding:

-- Decode an OTAP (BatchArrowRecords) file into the same flattened log schema
SELECT time_unix_nano, service_name, severity_text, body FROM read_otap_logs('https://github.com/smithclay/duckdb-otlp/raw/refs/heads/main/test/data/otap/logs-initial.bar');

Quickstart: Stream OpenTelemetry data

You can start a server that accepts OpenTelemetry data from instrumented code, AI agents such as Claude Code or Codex, or OpenTelemetry Collectors.

You can type some short commands in the DuckDB shell, run a Docker image that runs the extension as a daemon, or install the duckdb-otlp CLI with the install script or Homebrew.

-- See instructions above for loading otlp extension
-- use otap_serve() for OpenTelemetry Arrow
-- Listens for OTLP/HTTP on :4318 for metrics, logs, traces
FROM otlp_serve(
    'otlp:localhost:4318',
    token := 'dev-token-123456'
);

# Bootstraps an embedded DuckDB instance with the server running
# Writes data to a local DuckLake file
mkdir -p data 
export DUCKDB_OTLP_TOKEN=dev-token-123456

docker run --rm --name duckdb-otlp \
    -p 4318:4318 \
    -e DUCKDB_OTLP_TOKEN \
    -v "$(pwd)/data:/data" \
    ghcr.io/smithclay/duckdb-otlp:latest

# macOS or Linux, installs a single `duckdb-otlp` binary into ~/.local/bin
curl -fsSL https://smithclay.github.io/duckdb-otlp/install.sh | sh

# Listens for OTLP/HTTP on 127.0.0.1:4318 and OTLP/gRPC on 127.0.0.1:4317
# Writes data to a local DuckLake under ~/.local/share/duckdb-otlp
# Loopback-only with no token, so authentication is disabled automatically
duckdb-otlp serve

# macOS or Linux, installs a single `duckdb-otlp` binary
brew install smithclay/tap/duckdb-otlp

# Listens for OTLP/HTTP on 127.0.0.1:4318 and OTLP/gRPC on 127.0.0.1:4317
# Writes data to a local DuckLake under ~/.local/share/duckdb-otlp
# Loopback-only with no token, so authentication is disabled automatically
duckdb-otlp serve

duckdb-otlp query "SELECT time_unix_nano, service_name, severity_text, body FROM otlp_logs"

Send one hello-world log in OTLP/HTTP format with cURL:

curl -sS http://localhost:4318/v1/logs -H 'Authorization: Bearer dev-token-123456' -H 'Content-Type: application/json' -d '{"resourceLogs":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"curl-demo"}}]},"scopeLogs":[{"logRecords":[{"timeUnixNano":"1704067200000000000","severityText":"INFO","body":{"stringValue":"hello from curl"}}]}]}]}'

Query the data after ~5 seconds for the buffer to flush:

SELECT time_unix_nano, service_name, severity_text, body FROM otlp_logs;

Live ingest commits buffered rows in the background after about 5 seconds for the oldest buffered row or about 128 MiB of admitted request-body bytes. Use otlp_flush when readers need accepted rows durable and queryable while the server keeps running.

For a full walkthrough, including lakehouse ingest, see the docs.

Schema

The schemas align with a normalized version of the OpenTelemetry Arrow Data model as of extension release v0.5.0. Release v0.5.0 includes breaking schema changes from v0.4.0.

What You Can Do

• Read OTLP traces, logs, gauges, sums/counters, histograms, and exponential histograms from files.
• Read the columnar OpenTelemetry Arrow Protocol (OTAP) with the read_otap_* functions, which produce the same schemas as read_otlp_*.
• Stream live telemetry over OTLP (otlp_serve — OTLP/HTTP, or standard OTLP/gRPC unary with transport := 'grpc') or the OpenTelemetry Arrow Protocol (otap_serve — OTAP/Arrow bidirectional gRPC streaming) into the default DuckDB catalog, an attached DuckLake lakehouse, or an Iceberg REST catalog such as Amazon S3 Tables or Cloudflare R2 Data Catalog.
• Convert telemetry to Parquet files and save to cloud storage.
• Query local files, globs, S3, HTTP(S), Azure Blob, and GCS paths through DuckDB file systems.
• Use the browser demo for JSON, JSONL, and protobuf exploration with DuckDB-WASM: Interactive Demo.

Documentation

• Documentation Site
• Get Started
• Live Ingest Quickstart
• Stream to DuckLake
• Stream to Amazon S3 Tables
• Stream to Cloudflare R2 Data Catalog
• Store Claude Code or Codex Traces in Local DuckLake
• How-to Guides
• API Reference
• Schema Reference
• Live Ingest Reference
• Architecture

API at a Glance

The extension registers read_otlp_metrics and read_otlp_metrics_summary, but those functions remain unsupported until the project defines stable schemas for those shapes. See the API Reference for details.

Installation

The extension, inside any DuckDB:

INSTALL otlp FROM community;
LOAD otlp;

The duckdb-otlp CLI and receiver, as a standalone binary. Each release ships a tarball for linux-amd64, linux-arm64, darwin-amd64 and darwin-arm64, alongside SHA256SUMS; the container image holds the same binary. See the CLI Reference for the download-and-verify steps.

For source builds, development commands, and WASM builds, see CONTRIBUTING.md. WASM supports JSON, JSONL, and protobuf file reads, but not the live ingest server.

Limits

Early-stage and single-node (one daemon, one writer — no HA or horizontal scaling). Ingest has been benchmarked at ~100k logs/s on a 4-vCPU node; querying at volume is unproven, so test on your own data.

• Durability is the seal, not the 202. Live ingest buffers in memory and commits on a periodic group-commit ("seal"); a 202 means accepted, not durable. Call otlp_flush/otlp_stop before shutting down — a hard kill drops un-sealed rows (there is no WAL).
• Keep queries time-bounded. Data lands roughly time-ordered, so queries scoped by timestamp (and service_name) prune well; unbounded scans are slow. There is no full-text index — body substring/regex search and trace_id point lookups are brute-force scans: cheap over a short window, expensive over a wide one.
• File reads cap individual files at 100 MB. Live ingest runs via otlp_serve (OTLP/HTTP, or OTLP/gRPC unary with transport := 'grpc') and otap_serve (OTAP/Arrow gRPC streaming); both are native-only (not in the WASM build), bound request bodies, and apply max_buffered_bytes backpressure (503 / RESOURCE_EXHAUSTED); see the Live Ingest Reference.

Need Help?

• GitHub Discussions
• Open an issue
• Contributing guide

License

MIT. See LICENSE for details.