HUATUO 华佗

Kernel-wide Insight, Instant Observability, AutoTracing, Continuous Profiling

中文文档 · Quick Start · Documentation ·

---

What is HUATUO

HUATUO is a cloud-native operating system observability project open-sourced by DIDI and incubated under the CCF. It delivers kernel-level observability for general-purpose cloud-native computing, AI computing, and bare-metal infrastructure services. By integrating Linux kernel dynamic tracing technologies like kprobe, tracepoint, ftrace, and eBPF, HUATUO provides kernel-wide insights: finer-grained metrics, automatic context capture from kernel runtime, and intelligent tracing. Deployed at scale in Didi’s production environment, HUATUO plays a key role in troubleshooting system failures, enhancing the high availability and performance of cloud-native operating systems.

HUATUO is now listed in the CNCF Landscape.

Key Features

• Kernel-Wide Insight: Leverages BPF to maintain performance overhead below 1%, delivering full-stack, low-level observability insights into Linux kernel subsystems like MM, CPU scheduling, networking, and block I/O.
• Instant Observability: An event-driven runtime context capture mechanism that instruments kernel slow paths. It automatically triggers on events such as page faults, scheduling delays, generating detailed data for immediate analysis.
• AutoTracing: Employs automated snapshot retention to resolve performance jitters typical in cloud‑native and AI infrastructure environments, tackling issues such as CPU idle drops, CPU sys spikes, I/O surges, and Loadavg spikes.
• Continuous Profiling: A comprehensive and continuous performance profiling of the operating system and applications, covering CPU, Memory, I/O, and Locks. This drives business innovation and plays a key role in Chaos, HA and Stability Engineering.
• Heterogeneous Computing Hardware: Encompassing core system components such as CPUs, memory, PCIe interconnects, network adapters, and storage, alongside specialized AI accelerators including GPUs and NPUs.
• Ecosystem Integration: Integration with mainstream open-source observability stacks like Prometheus, Grafana, Pyroscope, and Elasticsearch. It automatically associates K8s container labels/annotations. Achieved through zero-instrumentation, kernel-level programming with eBPF, ensuring broad compatibility across hardware platforms and Linux distributions.

Big Picture

Ecosystem

Getting Started

• Quick Run

  To launch the HUATUO service with Docker:

    $ docker run --privileged --pid=host --cgroupns=host --network=host -v /sys:/sys -v /run:/run huatuo/huatuo-bamai:latest
  

  To pull metrics from another terminal:

    $ curl -s localhost:19704/metrics
  

• Quick Setup

  To launch the full stack (Elasticsearch, Prometheus, Grafana, and huatuo) using Docker Compose:

    $ docker compose --project-directory ./build/docker up
  

  Once running, access the monitoring dashboard at http://localhost:3000.

  

  

• NOTE

  Do not deploy images with the latest tag to production environments, as this is a development and testing image. Use a formal release image or binary.

Go Client

The public Go client is in client/. Node client files use the node_ prefix.

package main

import (
	"context"
	"encoding/json"

	nodeapi "github.com/ccfos/huatuo/apis/v1/node"
	"github.com/ccfos/huatuo/client"
)

func updateNodeConfig(ctx context.Context, address client.NodeAddress) error {
	nodeClient, err := client.NewNode(&client.NodeConfig{
		BearerToken: "node-token",
	})
	if err != nil {
		return err
	}
	return nodeClient.UpdateConfig(ctx, address, &nodeapi.UpdateConfigRequest{
		Config: map[string]json.RawMessage{
			"Runtime.CPULimitCores": json.RawMessage("1.5"),
		},
	})
}

Kernel Versions

The project supports kernel version 4.18 and later. The following kernel and OS distribution are primarily tested.

Documentation

For more information, visit https://docs.huatuo.tech

Co-Building

• ❇️ We welcome all users, developers, companies, and organizations to use Huatuo, report bugs, submit feature requests, share best practices, and help us build a professional and vibrant open-source community.
• ❤️ Contributors

Contact Us

• WeChat Group and Official Account:

License

This project is open source under the Apache License 2.0. The BPF code is licensed under the GPL license.