Systing

To build, ensure you have installed bpftool. This only builds on linux.

Previous versions of this tool had 3 distinct sub-commands, system, profile, and describe, as I was experimenting with different approaches to identifying problems. That code can be found in the old-systing branch.

The current iteration is just a single command, systing.

Quick start

To build, ensure you have installed bpftool. This only builds on linux.

cargo build
sudo ./target/debug/systing --duration 60

The BPF objects are compiled with -mcpu=v3 -fwrapv (pinned in build.rs) so a local build produces the same programs as a release build; set SYSTING_BPF_CLANG=/path/to/clang-21 to also use the release build's compiler version (clang 21), otherwise clang on PATH is used.

Alternatively, install directly with cargo. Use --locked so the build uses the dependency versions from Cargo.lock:

cargo install --locked --git https://github.com/josefbacik/systing.git

This will generate a trace.pb file which can be uploaded to a Perfetto instance for further analysis. Other --output extensions select other formats: .duckdb writes a queryable DuckDB trace database, and .systing (or .systing.gz) writes a lightweight profile export any tool can parse without DuckDB — see docs/PROFILE_EXPORT_FORMAT.md.

Shell completions

All three binaries can print their own completion script. systing uses a flag, the subcommand-based tools use a completions subcommand:

systing --completions bash
systing-analyze completions zsh
systing-util completions fish

Supported shells: bash, zsh, fish, elvish, powershell.

To install them for your user (bash and zsh shown):

# bash
mkdir -p ~/.local/share/bash-completion/completions
systing --completions bash > ~/.local/share/bash-completion/completions/systing
systing-analyze completions bash > ~/.local/share/bash-completion/completions/systing-analyze
systing-util completions bash > ~/.local/share/bash-completion/completions/systing-util

# zsh (any directory on your $fpath)
mkdir -p ~/.local/share/zsh/site-functions
systing --completions zsh > ~/.local/share/zsh/site-functions/_systing

Packagers can generate all files at once into a directory:

./scripts/generate-completions.sh completions/

Development Setup

IMPORTANT: If you're contributing code, enable the git hooks to enforce code formatting:

./setup-hooks.sh

This sets up automatic cargo fmt checks before commits and pushes. See CLAUDE.md for full development workflow details.

Running Integration Tests

Integration tests require root/BPF privileges and are marked as #[ignore] by default. Use the provided script to run them without causing build artifact ownership issues:

# Run all integration tests
./scripts/run-integration-tests.sh

# Run a specific test
./scripts/run-integration-tests.sh trace_validation test_e2e_parquet_validation

The script builds as your user (preserving artifact ownership), then runs only the test binary with sudo.

CI runs one of them on every push and pull request: the bpf-load-shapes workflow builds the bpf_load_shapes test binary with the release BPF compiler and runs its every_shape_loads gate in a guest per kernel (the repository's vmtest kernel and the Container-Optimized OS kernels pinned in .github/workflows/bpf-load-shapes.yml), so a program the verifier rejects at a shipping configuration fails the change before a tag.

Usage

Detailed options can be found here.

Enhanced Symbol Resolution

For improved symbol resolution, you can enable debuginfod support:

export DEBUGINFOD_URLS="https://debuginfod.fedoraproject.org/"
sudo ./target/debug/systing --enable-debuginfod --duration 60

This will fetch debug information from debuginfod servers, providing more accurate stack traces.

Recorder Management

Systing includes several recorders for different types of events. You can control which recorders are active using the following options:

List Available Recorders

sudo ./target/debug/systing --list-recorders

This will display all available recorders and their default states:

• sched - Scheduler event tracing (on by default)
• irq - IRQ and softirq event tracing (on by default)
• syscalls - Syscall tracing
• sleep-stacks - Sleep stack traces for all sleep states (on by default)
• interruptible-stacks - Interruptible sleep stack traces (on by default, requires sleep-stacks)
• cpu-stacks - CPU perf stack traces (on by default)
• network - Network connection state tracking
• network-syscalls - Network syscall-level tracing (send/recv bytes, retransmits, drops, stalls) without per-packet probes
• network-packets - Network packet-level tracing (sendmsg, recvmsg, qdisc, drops). On a large host this tier loses most of its events to the userspace consumer, so the packet tables are a sample: --packet-sample-rate N keeps 1 in N packets of the data-path event types (every stage of a kept packet is kept together, so per-stage latencies stay pairable; packet and byte counts scale by N; the diagnostic events — zero-window probes, RTO timeouts, drops, state changes — are never sampled), the rate ran is recorded as sysinfo.network_packet_sample_rate, and the exit summary prints the events missed beside the events recorded.
• memory - Memory usage tracking (RSS, mmap/munmap/brk, page faults; host-wide THP/compaction counters into memory_vmstat). --memory-vfio adds VFIO DMA regions and the IOMMU map/unmap run-size histogram (how fragmented the memory behind device mappings is); --memory-thp-sample-rate N adds sampled THP-split events with stacks. Both legs turn themselves off (named in sysinfo) on hosts without the symbols.
• memory-alloc - Heap allocator uprobes (malloc/calloc/realloc/free) with stacks
• markers - Userspace marker events (faccessat2 with mode=-975)
• tpu - TPU profiling (gRPC to XLA runtime profiler service)
• tpu-metrics - TPU runtime metrics polling (port 8431, always available)
• task-stacks - Periodic stack snapshots of every targeted thread, blocked ones included (see Task Stacks)

The three network* recorders are tiers of the same subsystem, ordered by event volume — see Network Traffic Recording below for when to use each.

Python stack symbolization is not a recorder; enable it with --collect-pystacks, which resolves Python frames in whichever stacks the active recorders collect. In a stack, the Python frames stand where the interpreter ran them among the native frames: each run of Python frames takes the place of the _PyEval_EvalFrameDefault frame it executed in, so C called from Python, and Python called back from C, read in order. When the two cannot be paired up (a native stack that lost an interpreter loop frame, because the frame-pointer unwinder skips the caller of a function built without frame pointers, or Python 3.11 re-entered from C, which leaves no marker of where) the Python frames come first as one block, followed by the native ones. That is decided stack by stack, so one Python function can appear under both shapes in a trace. The interpreter's own entry frames (3.12's <interpreter trampoline>) are not shown.

--collect-build-id captures user stacks as (build-id, file offset) pairs instead of raw addresses, so frames of processes that exit before end-of-trace symbolization still resolve — through a build-id-keyed store filled from /usr/lib/debug/.build-id, debuginfod (with --enable-debuginfod), and the binaries of still-running processes. Frames whose build-id no source knows render as unknown ([buildid:<hex>]) <0x<offset>>, a stable identity that can be resolved offline later. Costs ~40% larger stack-ring reservations while enabled; off by default and free when off. The still-running-processes source is an end-of-trace walk over every live process's executable mappings — the sampled processes first, then the rest newest first (by descending pid, a heuristic for age: a young process holding a recycled low pid after the pid space wraps walks last) — that reads each distinct file's build-id note once: a file is recognised across processes by the maps line's dev/inode and, across mounts (one container image's file shows a different device in every container), by its mapped path, size and mtime from one stat. The walk is bounded by --build-id-index-max-files (default 10000; 0, or more than the store keeps, means the store's capacity of 16384) and --build-id-index-max-ms (default 2000, checked between file reads; 0 = unbounded); it prints a build-id index: … line with what it read, how much each tier deduplicated, and whether a bound stopped it. A binary the walk never reaches — a bound stops it at the oldest processes — still resolves through the other sources or keeps its [buildid:<hex>] identity.

Note: sleep-stacks acts as a master switch for all sleep stack collection. When enabled, both uninterruptible (D state) and interruptible (S state) sleep stacks are collected by default. Use --no-interruptible-stack-traces or disable the interruptible-stacks recorder to collect only uninterruptible sleep stacks.

Add Specific Recorders

Use --add-recorder to enable additional recorders on top of the defaults:

# Enable syscalls in addition to default recorders
sudo ./target/debug/systing --add-recorder syscalls --duration 60

# Enable network traffic recording in addition to default recorders
sudo ./target/debug/systing --add-recorder network --duration 60

# Enable multiple additional recorders
sudo ./target/debug/systing --add-recorder syscalls --add-recorder network --duration 60

Use Only Specific Recorders

Use --only-recorder to disable all recorders and enable only the ones you specify:

# Only record syscalls (disable everything else)
sudo ./target/debug/systing --only-recorder syscalls --duration 60

# Only record network connection state (disable everything else,
# including the packet-level probes — see Network Traffic Recording)
sudo ./target/debug/systing --only-recorder network --duration 60

# Only record syscalls and cpu-stacks
sudo ./target/debug/systing --only-recorder syscalls --only-recorder cpu-stacks --duration 60

Network Traffic Recording

The network recorder captures detailed network traffic information including:

• TCP and UDP send/receive operations at the connection level
• Packet-level latency tracking through the network stack
• Timing information for packet transmission and reception
• Queue latencies and buffer management

Note: Network recording is disabled by default to minimize overhead. Enable it explicitly when you need to analyze network performance.

Network tracing is split across three recorders, ordered by event volume: network tracks TCP connection state (socket lifecycle and state transitions via inet_sock_set_state); network-syscalls adds per-syscall send/receive accounting plus the low-frequency diagnostics (retransmit timer, zero-window probes, sndbuf stalls, packet drops) — bytes, stalls and drops per connection at syscall-rate cost; and network-packets adds the per-packet and per-poll probes (transmit/receive path, qdisc, epoll), whose event volume is bounded by traffic: on a large host the userspace consumer falls behind and the exit summary reports the events it missed, so keep a packets capture short or run it with --packet-sample-rate N. --add-recorder network enables state + packets, the usual shape for an investigation. --only-recorder enables exactly what you name (each tier pulls in the base network recorder it requires), so --only-recorder network is state-only and --only-recorder network-syscalls is the shape for continuous or fleet-wide profiling, where per-packet volume is prohibitive but per-connection throughput still matters.

# Enable network recording (connection state + packet-level)
sudo ./target/debug/systing --add-recorder network --duration 60

# Connection state only — no packet-level probes
sudo ./target/debug/systing --only-recorder network --duration 60

# Connection state + syscall accounting + retransmit/drop/stall diagnostics
sudo ./target/debug/systing --only-recorder network-syscalls --duration 60

# Full network tracing and nothing else
sudo ./target/debug/systing --only-recorder network-packets --duration 60

The network recorder instruments multiple points in the Linux network stack:

• tcp_sendmsg/udp_sendmsg - Connection-level send operations
• tcp_recvmsg/udp_recvmsg - Connection-level receive operations
• __tcp_transmit_skb/udp_send_skb - Packet transmission
• tcp_rcv_established/__udp4_lib_rcv - Packet reception
• __dev_queue_xmit/net_dev_start_xmit - Device queue and transmission
• And additional points for tracking packet flow through queues and buffers

Task Stacks

The task-stacks recorder snapshots the stack of every targeted thread (--pid, --cgroup or the traced command; every thread on the host without one) every --task-stacks-interval-ms (default 100), with a sleepable BPF task iterator. A trace then shows what each thread was doing, blocked ones included, not only what was on a CPU. It needs Linux 6.2 or newer.

sudo systing --add-recorder task-stacks --task-stacks-frames all --pid 1234 -d 10

• --task-stacks-frames picks the frames: native (kernel frames, and native user frames unwound by frame pointers), python (Python frames alone, and only the threads that have any) or all. python and all turn --collect-pystacks on; when the option is not given it is all with --collect-pystacks and native without.
• With -d the recorder takes ceil(duration / interval) snapshots, numbered from 1 (10 s at 100 ms: iterations 1-100).
• A thread that has not run since its last snapshot and is still in the same non-runnable state cannot have changed its stack: it is not walked again and its row is extended instead, so a thread blocked for a minute is one row.
• The rows are the task_stack_event table (SCHEMA_CHANGES.md, schema 23), with the stack by stack_id into stack like every other recorder's. With Python frames collected, the thread table also gets the name the process gave each thread (thread.py_name: threading.Thread(name=...)), read out of the interpreter: Python 3.13 and 3.14 for now.
• In the Perfetto trace each thread gets a Task Stacks: <thread> track, titled with the names that go with the frames asked for: the kernel's name for the thread (comm) with native, the Python name with python, both with all (Task Stacks: MainThread [python3]). A thread that has no Python name, another language's or an older Python's, goes by the kernel's. The track is the stack over time the way py-spy's Chrome trace output draws it, each frame one slice for as long as it stays on the stack, root at the top. A slice is named after the function alone; language, file (the full path, for Python frames and for native frames with debug info), line, module and address are its arguments.

What to keep in mind: a row's ts is the start of its iteration's walk, which reaches a given thread up to one walk later. A blocked thread's stack is exact; a running thread's is read while it runs and is a best effort. The unchanged-thread skip and the CPU-time deltas hold for up to 65,536 targeted threads. --collect-build-id does not apply to these stacks. The recorder keeps its events until the capture ends, so it cannot be used with --continuous.

Debugging and Verbosity

Use multiple -v flags to control verbosity levels:

# Basic informational output
sudo ./target/debug/systing -v --duration 60

# Detailed debugging (useful for troubleshooting)
sudo ./target/debug/systing -vv --enable-debuginfod --duration 60

# Maximum verbosity (includes library debugging)
sudo ./target/debug/systing -vvv --enable-debuginfod --duration 60

This tool traces all the scheduling events on the system, cgroup, or process and generates a Perfetto trace. This can be uploaded to a local perfetto instance for further analysis, or you can use the public one here.

NOTE: With cgroup and process tracing, you will see other processes that appear to not end, this is because the tool is tracing the scheduling events captures the process going off the CPU or going on the CPU in addition to the process being traced, so you will miss events for the unwanted process leaving the CPU. Perfetto handles this appropriately, but it looks odd.

--cgroup <path> traces every task whose cgroup is that cgroup or any cgroup below it, including cgroups created after the trace started — membership is decided by the kernel's bpf_task_under_cgroup(), which needs Linux kernel 6.5 or newer. On older kernels (no bpf_task_under_cgroup in the kernel's BTF) systing falls back to matching a snapshot of the target's cgroups taken when the trace starts, so cgroups created under the target afterwards are not traced; the mode in use is printed at start. Setting SYSTING_CGROUP_FILTER_LEGACY to any non-empty value (=1 will do) forces the fallback on any kernel. Kernels before 6.6.117 / 6.12.58 / 6.18 resolve the target within systing's own cgroup namespace (upstream commit 2c8951339506 lifted that limit): there, from a container with a private cgroup namespace and the host's cgroup filesystem mounted, a host path is visible but cannot be resolved, and systing says so at start — run it in the host cgroup namespace or use the fallback. Only the unified cgroup v2 hierarchy is supported; a target on a cgroup v1 hierarchy, or a directory that is not on a cgroup filesystem at all, is refused at start with that cause named (systing checks the target's filesystem before blaming the namespace).

--trace-event - This will add an instant track event for each event that this tool captures. The format is ":::". This is most easily obtained by running

bpftrace -lp <pid of desired program> | grep <name of usdt>

The currently allowed formats are

• usdt:/path/to/executable:tracepoint_name:tracepoint_class
• uprobe:/path/to/executable:function_name
• uprobe:/path/to/executable:offset
• uprobe:/path/to/executable:function_name+offset
• uretprobe:/path/to/executable:function_name
• uretprobe:/path/to/executable:offset
• uretprobe:/path/to/executable:function_name+offset
• kprobe:function_name
• kprobe:offset
• kprobe:function_name+offset
• kretprobe:function_name
• kretprobe:offset
• tracepoint:subsystem:tracepoint_name

For all usdt and u*probe events you must specify --trace-event-pid to to indicate which PID's you wish to record the events for. For example, if you want to trace when qemu does a v9fs create, you would run the following

systing --trace-event-pid <PID of qemu> --trace-event "usdt:/usr/bin/qemu-system-x86_64:qemu:v9fs_create"

Custom track events

You can also add complex track event configurations to the trace. Examples of these configuration files can be found in the examples directory.

📖 For complete documentation on the JSON configuration format, see docs/TRACE_CONFIG_FORMAT.md

The format is a JSON file specified with --trace-event-config.

The pthread_mutex example will add a track that shows the time spent locking the mutex and the time that the mutex is locked by the thread.

{
  "events": [
    {
      "name": "mutex_entry",
      "event": "usdt:/usr/lib64/libc.so.6:libc:mutex_entry",
      "args": [
        {
          "arg_index": 0,
          "arg_type": "long",
          "arg_name": "mutex_addr"
        }
      ]
    },
    {
      "name": "mutex_acquired",
      "event": "usdt:/usr/lib64/libc.so.6:libc:mutex_acquired",
      "args": [
        {
          "arg_index": 0,
          "arg_type": "long",
          "arg_name": "mutex_addr"
        }
      ]
    },
    {
      "name": "mutex_release",
      "event": "usdt:/usr/lib64/libc.so.6:libc:mutex_release",
      "args": [
        {
          "arg_index": 0,
          "arg_type": "long",
          "arg_name": "mutex_addr"
        }
      ]
    },
  ],
  "tracks": [
    {
      "track_name": "pthread",
      "ranges": [
        {
          "name": "locking",
          "start": "mutex_entry",
          "end": "mutex_acquired"
        },
        {
          "name": "locked",
          "start": "mutex_acquired",
          "end": "mutex_release"
        }
      ]
    },
  ]
}

The args field is optional and allows you to capture probe arguments that will show up as debug annotations on the events in the trace. Up to 4 args can be captured per event. Each arg specifies:

• arg_index: Which argument to capture (0-based index)
• arg_type: The type of the argument ("string", "long", or "retval")
• arg_name: The name of the debug annotation (e.g., "mutex_addr")

Available arg types:

• "string": Captures a string pointer argument (requires arg_index)
• "long": Captures a 64-bit integer argument (requires arg_index)
• "retval": Captures the function return value (only valid for kretprobe and uretprobe; arg_index not used)

These debug annotations provide additional context when viewing the trace in Perfetto, showing the captured value with the specified name. In the example above, the mutex address will appear as a "mutex_addr" annotation on each event.

Kernel Version Requirements

Important: Capturing arguments from tracepoint events requires Linux kernel 6.10 or newer.

Prior to kernel 6.10, the BPF raw_tracepoint infrastructure did not support bpf_get_attach_cookie(), which is required for systing to capture arguments from tracepoint events. This limitation affects:

• tracepoint:syscalls:sys_enter_* events (e.g., sys_enter_mmap, sys_enter_open)
• tracepoint:syscalls:sys_exit_* events
• Any other custom tracepoint events with arguments

What this means:

• ✅ Kernel 6.10+: Full support for tracepoint argument capture
• ⚠️ Kernel < 6.10: Tracepoint events work, but argument capture is not supported

If you attempt to use tracepoint argument capture on kernel < 6.10, systing will fail with a clear error message:

Cannot capture tracepoint arguments on kernel < 6.10.
Tracepoint 'syscalls:sys_enter_mmap' has 2 argument(s) configured, but this kernel
version doesn't support bpf_get_attach_cookie() for raw_tracepoint programs.
Either upgrade to kernel 6.10+ or remove the argument specifications from the event configuration.

Other probe types are not affected:

• kprobe/kretprobe - Work on all kernel versions
• uprobe/uretprobe - Work on all kernel versions
• usdt - Work on all kernel versions

To check your kernel version:

uname -r

For kernel < 6.10, you can still use tracepoint events without arguments, or use kprobes/uprobes instead.

The stack field is optional (defaults to false). When set to true, systing will capture and emit a stack trace whenever this event fires. This allows you to see the call stack at the point where the event occurred, which is useful for debugging and performance analysis.

This results in a track that looks like this