otelcol-genai-sketches

An OpenTelemetry Collector distribution for continuous, bounded answers about high-cardinality agent traffic without exporting or indexing every underlying value. It turns GenAI traces into bounded Prometheus metrics and keyed top-k summaries.

Use it alongside an existing trace backend to find where reported token volume is accumulating, measure missing usage, and keep high-cardinality identities out of metric labels.

Start With The Question

These are fleet-level measurement signals. Trace explorers and evaluation systems remain the right tools for understanding one agent run or judging its output.

Quick Start

Requirements: Docker with Compose, Go 1.26.6 or newer, and openssl.

git clone https://github.com/llm-measurement/otelcol-genai-sketches.git
cd otelcol-genai-sketches
export GENAI_SKETCH_SECRET="$(openssl rand -hex 32)"
make example-up

This starts a sample application, the collector, a Prometheus metrics server, and a provisioned Grafana dashboard.

• Grafana: http://localhost:3000
• Prometheus: http://localhost:9090
• Collector metrics: http://localhost:8889/metrics

The sample emits model, agent, tool, and retrieval spans. It also includes missing token fields and enough prompt variety to exercise bounded estimates.

Get A Useful Result

Let the example run for at least one minute, then use the dashboard in this order:

1. Compare Requests/sec with Reported Token Rate.
2. Check Reported Tokens / Request to separate traffic growth from larger requests or responses.
3. Check Missing Token Usage before treating token totals as complete.
4. Compare model and team/model slices to localize the change.

Then inspect the high-cardinality surface:

docker compose -f examples/compose.yaml logs collector \
  | grep 'genaisketch topk snapshot'

The snapshot contains keyed hashes, estimates, and lower and upper bounds. It does not contain prompt text, and its hashes never become Prometheus labels.

Stop the example with:

make example-down

The token-consumption playbook contains the PromQL queries and an interpretation table for the same workflow.

Keep Your Current Backend

You can add the connector without replacing Datadog, Langfuse, Alloy, or another OTLP destination:

applications -> Collector fan-out -> current trace backend
                                  -> bounded sketch metrics

The distribution includes OTLP gRPC and HTTP exporters. CI verifies that one trace batch can be forwarded while the connector derives metrics from it. You can also let an existing Collector or Alloy deployment own the fan-out and run this distribution as a sidecar.

See Keep Your Existing Telemetry Backend for tested generic OTLP configurations and coexistence paths for an ordinary Collector, Datadog, Langfuse, and Grafana Alloy.

When This Fits

Use this collector when you need to:

• keep metric cardinality bounded across a large GenAI or agent workload;
• estimate distinct users, prompt signatures, or retrieval documents without placing raw values in aggregate state;
• separate real zero-token usage from requests that omitted token attributes;
• investigate unexpected or runaway token consumption, sometimes called "token maxing," by locating where reported tokens accumulate;
• inspect token-heavy prompt signatures without turning them into labels; or
• count model requests without including agent, tool, retrieval, workflow, and MCP spans in the same denominator.

This is not a prompt logger, billing ledger, arbitrary attribute-to-label converter, anomaly detector, loop stopper, budget enforcer, or differential-privacy system.

If exact traces are safe to retain and remain fast and affordable to query, use them. The connector is an always-on bounded evidence surface, not a replacement for raw records needed for diagnosis, audit, or replay.

What It Produces

Optional MCP metrics estimate distinct sessions, methods, and resources. Weighted top-k prompt signatures are emitted as structured logs with estimates and lower and upper bounds. They never become Prometheus labels.

See Metrics for exact signal semantics.

How It Fits

GenAI applications -> OTLP traces -> this collector -> Prometheus metrics
                                      |              -> bounded structured logs
                                      +--------------> optional existing OTLP backend

Use this distribution when source spans already flow through OpenTelemetry. If you own a custom streaming, batch, or warehouse pipeline and do not need OTLP-to-metrics conversion, use llm-sketchkit directly.

The connector uses llm-sketchkit for canonicalization, keyed hashing, distinct counting, frequent-item estimates, and deduplication. Raw prompt text, user IDs, document IDs, and request IDs do not enter connector aggregate state or its derived metrics and snapshots.

An optional forwarded trace remains the original trace. If instrumentation captured raw content, the existing trace backend still receives it. See the shadow-mode guide before enabling fan-out.

Use In An Existing Collector

The connector is also published as a standalone Go module for the OpenTelemetry Collector Builder. Add it to a builder manifest:

connectors:
  - gomod: github.com/llm-measurement/otelcol-genai-sketches/connector/genaisketchconnector v0.1.0-alpha.1

Configure genaisketch as an exporter from the traces pipeline and a receiver in the metrics pipeline. The path: override in this repository's builder manifest exists only for a local checkout.

Configuration

Start with the example configuration. The connector requires a secret of at least 16 bytes from GENAI_SKETCH_SECRET by default.

connectors:
  genaisketch:
    window_duration: 1m
    retention_windows: 10
    max_slices: 2000
    topk: 20
    slices:
      - name: model
        keys: [gen_ai.request.model]
        from_resource_attributes: [gen_ai.request.model]

Slice values are exported in cleartext as Prometheus labels. Use only bounded, low-cardinality, non-sensitive attributes such as model, team, route, or provider. Configured slice capacity uses deterministic inactive-slice eviction and one __overflow__ value. Excess traffic is counted rather than silently dropped, and it does not create new label values.

See Configuration for field mapping, operation filtering, resource fallback, MCP support, deduplication, and capacity limits.

Security And Privacy

Keyed hashes are pseudonymous, not anonymous. Values remain linkable while the same secret is in use, and anyone holding the secret can test candidate values. Rotating the secret breaks comparison with earlier windows.

The structured top-k surface contains keyed hashes and bounded estimates. Treat collector logs as sensitive operational data even though raw source values are not included. The connector rejects known high-cardinality MCP identifiers as slice keys and rejects overlap between plaintext slice keys and configured hashed fields.

Token attributes are optional. Missing usage is counted explicitly; the connector does not invent token weights. Bloom-filter deduplication is bounded and may undercount because false positives are possible.

See Security to report a vulnerability privately.

Evidence

Recorded local measurements include:

• 36 million spans accepted and exported over 60 minutes at 10,000 spans/second;
• exact request filtering across a 36 million-span mixed tree workload: 10.8 million emitted model spans and 10.8 million counted requests;
• exact missing-usage accounting for 1.08 million planted missing-token model spans;
• 528,924 spans/second in the mixed in-process benchmark; and
• 1,386.8 MiB maximum collector RSS in the mixed fleet-shaped soak.

These are measurements from one Apple M4 Max system, not universal capacity claims. Workload definitions, machine details, commands, and non-passing runs are in Benchmarks.

Development

make tidy
make check
make dist
make test-integration

The integration suite covers OTLP-to-Prometheus behavior, gRPC and HTTP shadow-mode fan-out, bounded overflow, deterministic eviction, restart stability, tree locality, and sentinel scans across metric, label, and structured-log surfaces.

Status

This project is alpha software. Interfaces and metric semantics may change between alpha releases. Pin an exact version and test it against your own traffic before production use. See the changelog for release notes.

Licensed under the Apache License 2.0.