vulnemetry

Currently, most Go programs have vulnerabilities scanned in CI, or even have their images scanned in production. GoVulncheck allows you to find out which function you are using, coupled with which package has a vulnerability, but there is no way to find this out for a live, running system in production.

This project aims to experiment converting vulnerability scanning from a discrete process to a continuous observability system, which allows you to scan any Go service in your cluster. By importing a library to report the code executing, and then matching it against the vulnerability database that continuously updates itself. This gives you both code that is vulnerable and the fact that it is executing right now. As set of metrics in a Prometheus native fashion, which you can then create alerting and visibility for.

NOTE [EXPERIMENTAL] This is still an experimental project aimed at showcasing the feasibility of this aspect and is not to be used in production applications yet, since there is much future work to be researched and implemented.

Running the demo via docker compose

make up

Grafana at localhost:3000 (admin/admin) showcases observability of the vulnerabilities executing based on your service and others.

Applying it to your own code

    vprofiler, err := vulnprofiler.New(vulnprofiler.Config{
		Service: "checkout",
		Window:  10 * time.Second,
	})
	if err != nil {
		log.Fatal(err)
	}

	go vprofiler.Run(ctx)

	mux := http.NewServeMux()
	mux.Handle("GET /vulnexec", vprofiler.Handler())

How the system works

Go Service Your Go service imports a library that basically runs the profiler and debug info at regular intervals of time. This is then used to determine how often frames or parts of your code are executing. It is then exposed, similar to Prometheus metrics, at different time points, showcasing that these are the modules running of this version and this is the code running which has executed this many times.

OpenTelemetry The OpenTelemetry Collector is now used to regularly pull from your set of services and push it into the vulncollector, so it handles the transport from your services to the defined collector.

Vulncollector Similar to other Prometheus collectors, vulncollector receives, for your service, the frames of the programs that have occurred. It tries to translate the profiler output into a version that is compatible with or similar to Govulncheck's database naming of functions that are vulnerable. It also keeps a copy of the vulnerability database and, at regular intervals, updates it. It then performs this join of the two, which gets you both code that is vulnerable and how many times it's been executed, which it then uses to expose Prometheus metrics

Observability Prometheus then takes in the data from the vulncollector and stores it, which is then, in the case of our demo, used by Grafana to visualize in real time:

• The clusters' vulnerabilities
• How many services are affected
• How vulnerabilities change with releases
• The vulncollector's self-observability as well

vulnprofiler/   <- import this in your service. samples it, SERVES a counter table
vulnmatch/      <- import this in if you want to build your own collector
vulncore/       <- the one contract both sides must agree on
vulncollector/  <- An example shipped collector
e2e/            <- everything that checks this system, incl. the deliberately-
                  vulnerable programs being observed
demo/           <- compose + config for the demo stack

What is exposed

The producer, on the route you mount it at. vulnexec_frame_windows_total and vulnexec_frame_samples_total carry the frame labels symbol · package · module · version · observer; the two info gauges carry service · replica · boot_id · goos · goarch · cpu_profiling.

The collector, on its ops surface. The first two carry the full fleet key service · replica · osv_id · symbol · package · module · version · severity · match_kind · observer which is affordable only because the cross-reference already happened:

Future work

• Translation Currently, converting from profiler output to understanding that this is the vulnerable code is, let's say, not an optimal process, and we currently cannot have all of the executing code visible. Work should still be done to understand how to probe and transport these values better, possibly using eBPF or compiler options, etc.
• Security Hardening Right now, the module versions and executing code are exposed as a metric, which can be isolated using network configuration, but we'd like to add things like authentication so that only the collector can read from this, or even alternate formats that won't expose these values.
• Distributed systems availability and performance Right now, there exists only vulncollector and one Postgres instance to service it, but we still need to test for multiple collectors existing within a node to make infrastructure ops easier to reason about availability. Also, the profiler and collector are running as lightweight as possible so as not to have much overhead.