OpenTelemetry eBPF Profiler

This repository implements a whole-system, cross-language profiler for Linux via eBPF.

Core features and strengths

• Implements the Alpha OTel Profiles signal
• Very low CPU and memory overhead (1% CPU and 250MB memory are our upper limits in testing and the agent typically manages to stay way below that)
• Support for native C/C++ executables without the need for DWARF debug information (by leveraging .eh_frame data as described in US11604718B1)
• Support profiling of system libraries without frame pointers and without debug symbols on the host.
• Support for mixed stacktraces between runtimes - stacktraces go from Kernel space through unmodified system libraries all the way into high-level languages.
• Support for native code (C/C++, Rust, Zig, Go, etc. without debug symbols on host)
• Support for a broad set of HLLs, like Hotspot JVM, Python, Ruby, PHP, Node.JS, V8, Perl, Erlang and .NET.
• 100% non-intrusive: there's no need to load agents or libraries into the processes that are being profiled.
• No need for any reconfiguration, instrumentation or restarts of HLL interpreters and VMs: the agent supports unwinding each of the supported languages in the default configuration.
• ARM64 support for all unwinders.
• Support for native inline frames, which provide insights into compiler optimizations and offer a higher precision of function call chains.

Supported Linux kernel version

The minimum required Linux kernel version has increased with certain commits. Specifically:

• Commit 8047150e was the last to support kernel version 5.4. Subsequent changes may require a minimal Linux kernel version of 5.10 or greater.
• Commit 7ddc23ea was the last to support kernel version 4.19. Subsequent changes may require a minimal Linux kernel version of at least 5.4.

Updating the supported Linux kernel version

The project maintains its minimum supported kernel version in line with the lowest kernel version currently provided by actively maintained major Linux distributions, which include Debian stable, Red Hat Enterprise Linux, Ubuntu LTS, Amazon Linux and SUSE Linux. The minimum requirement may be increased when all such distributions no longer ship a specific kernel version. This approach enables the codebase to utilize newer eBPF features and avoids the need to maintain compatibility shims for obsolete kernels.

It should be noted that certain distributions incorporate eBPF features from newer kernels into their supported versions. When this occurs, the distribution's stated kernel version does not accurately reflect its true eBPF capabilities and will not prevent us from increasing the minimum supported version. On such kernels, the no-kernel-version-check configuration option can be used to bypass the checks and allow the profiler to execute.

Building

We have integrated the profiler into the OTel Collector as a receiver, and this is the supported configuration going forward.

To aid with development, testing and debugging, we also offer a standalone profiling agent binary named ebpf-profiler, and a local build of an OTel Collector profiling receiver binary (otelcol-ebpf-profiler). These binaries are not supported in any way, can be dropped in the future and should not be deployed in production.

Platform Requirements

The agent can be built with the provided make targets. Docker is required for containerized builds, and both amd64 and arm64 architectures are supported.

For Linux, the following steps apply:

1. Build the agent for your current machine's architecture:

   make agent
   

2. To cross-compile for a different architecture (e.g. arm64):

   make agent TARGET_ARCH=arm64
   

The resulting binary will be named ebpf-profiler in the current directory.

Other OSes

Since the profiler is Linux-only, macOS and Windows users need to set up a Linux VM to build and run the agent. Ensure the appropriate architecture is specified if using cross-compilation. Use the same make targets as above after the Linux environment is configured in the VM.

Alternative Build (Without Docker)

You can build the agent without Docker by directly installing the dependencies listed in the Dockerfile. Once dependencies are set up, simply run:

make

This will build the profiler natively on your machine.

Building otelcol-ebpf-profiler locally (Without Docker)

You can build the local otelcol-ebpf-profiler binary by running:

make otelcol-ebpf-profiler

or to cross-compile for a different architecture (e.g. arm64):

make otelcol-ebpf-profiler TARGET_ARCH=arm64

See local.example.yaml for an example configuration.

Running

You can start the agent with the following command:

sudo ./ebpf-profiler -collection-agent=127.0.0.1:11000 -disable-tls

To start the OTel Collector profiling receiver, run:

sudo ./otelcol-ebpf-profiler --feature-gates=+service.profilesSupport --config cmd/otelcol-ebpf-profiler/local.example.yaml

The agent comes with a functional but work-in-progress / evolving implementation of the recently released Alpha OTel Profiles signal.

The agent loads the eBPF program and its maps, starts unwinding and reports captured traces to the backend.

Open Source Backends

As the OTel Profiles signal is still in development, mature production-ready backends have yet to emerge. The following open source projects can be used as backends:

• devfiler — to speed up development and experimentation, Elastic has open-sourced a desktop application that reimplements the backend (collection, data storage, symbolization and UI) portion of the eBPF profiler. Note that devfiler is not a real production backend and should not be used as such. It is solely aimed at testing, experimentation and development.
• Pyroscope — an open source continuous profiling database that natively supports ingesting OTel profiling data.

Contributing

We welcome contributions of all kinds: bug reports, feature requests, documentation improvements and code. Everyone is welcome to open issues and send pull requests.

Before contributing, please read the contributing guidelines. They cover the development prerequisites, how to build and test your changes, the pull request workflow, and how PRs get reviewed and merged.

To understand how this project works and learn more about profiling, check out Profiling internals.

This project is developed by the OpenTelemetry Profiling SIG, which meets regularly. See the OpenTelemetry community repository for the meeting schedule, and join us in the #otel-profiles channel on the CNCF Slack for discussions and questions.

Maintainers

• Felix Geisendörfer, Datadog
• Christos Kalkanis, Elastic
• Timo Teräs

For more information about the maintainer role, see the community repository.

Approvers

• Roger Coll, Elastic
• Florian Lehner, Elastic
• Damien Mathieu, Elastic
• Tim Rühsen

For more information about the approver role, see the community repository.

Emeritus

• Dmitry Filimonov, Maintainer
• Joel Höner, Approver

For more information about the emeritus role, see the community repository.

Become an Approver or a Maintainer

See the community membership document in OpenTelemetry community repo.

Legal

Licensing Information

This project is licensed under the Apache License 2.0 (Apache-2.0). Apache License 2.0

The eBPF source code is licensed under the GPL 2.0 license. GPL 2.0

Licenses of dependencies

To display a summary of the dependencies' licenses:

make legal