Add new ebpf integration

[tpaschalis]

PR Description

This PR adds a v2 integration based on cloudflare's ebpf_exporter.

One limitation of the specific exporter is that it doesn't support uprobes/uretprobes. While I was able to fork the exporter and patch in support for those, I thought we'd keep this simple for the time being and create a first version which worked as-is.

The user configures a number of ebpf "programs", which are then attached to the correct place in the kernel. The exporter reads and decodes values from an eBPF map, exposing them as Prometheus metrics.

integrations:
  ebpf:
    programs:
    # Count timers fired in the kernel
    - name: timers
      metrics:
        counters:
          - name: timer_start_total
            help: Timers fired in the kernel
            table: counts
            labels:
              - name: function
                size: 8
                decoders:
                  - name: ksym
      tracepoints:
        timer:timer_start: tracepoint__timer__timer_start
      code: |
        BPF_HASH(counts, u64);

        // Generates function tracepoint__timer__timer_start
        TRACEPOINT_PROBE(timer, timer_start) {
            counts.increment((u64) args->function);
            return 0;
        }

Which issue(s) this PR fixes

No issue, this came out of the Hackathon :)

Notes to the Reviewer

If you agree with the general premise and idea, I can start working on adding documentation, tests, etc.

Do you think there should be behind an additional ebpf_enabled build tag? The current PR fails failed because it couldn't build the gobpf dependency on the Linux image that we run golangci-lint on. I don't think we should add ebpf/bcc bindings into our linting step, just to stop it from complaining, so an extra build tag might be a solution here.

I thought we'd start with a v2 integration outright, as a way to encourage more people to start using them. How do you feel about this?

PR Checklist

• CHANGELOG updated
• Documentation added
• Tests updated

[rfratto]

IMO for now I'd ready have stuff like this be opt out (i.e., the default behavior is to build everything and you use tags to disable specific things).

Can the tag be !noebpf? We use nodocker and nonetwork for disabling specific types of tests.

[tpaschalis]

The issue which prompted the opt-in tag is that the Linux agent builds would fail unless some specific system packages are installed (due to the Go->eBPF bindings dependency, which in turn requires specific kernel versions).

$ go build .          # on linux system without support for bcc
# github.com/iovisor/gobpf/bcc
../../vendor/github.com/iovisor/gobpf/bcc/module.go:32:10: fatal error: 'bcc/bcc_common.h' file not found
#include <bcc/bcc_common.h>
         ^~~~~~~~~~~~~~~~~~
1 error generated.

I'm not sure what's an elegant way to handle this. Bundling the required C headers, and somewhat complicating our build pipeline?

[rfratto]

You can update tools/seego/Dockerfile (it has apt, it's Debian-based) to install dependencies. We do this already in the main Dockerfile to install systemd headers, and extending the build image to include our specific changes would be the way to do this IMO.

I think we can assume the kernel version is new enough, the platforms we'll be mainly building on should have kernels newer than ones from 2017.

[tpaschalis]

I see. We'll also need to add the dependencies when linting.

My main concern here is that people who build from source on Linux, might be confused when go build fails with that cryptic message. On one hand, they would already need the systemd headers as you mentioned, but I'm not sure that bcc is also something they'd expect to need if they're not using eBPF.

I'll try to find a way to make it work and report back :)

[rfratto]

That's fair :) We can document it in our contribution guide (we should probably be mentioning systemd there too but we're not at the moment). Do you think that would be helpful enough?

[rfratto]

If we're keeping track of build dependencies in the contribution guide we should probably make sure we link to that from our README, too.

[tpaschalis]

Yeah, that would make sense ^^

[tpaschalis]

I added a small note about these two dependencies on our contribution guide 323fb00.

Do we still want to have a noebpf flag to make it easier for people to contribute?

[rfratto]

LGTM, the integration seems fine % my nits and comment about finding a way to build this by default in our CI setup.

[rfratto]

Will it cause any problems if two of these are created? We should make sure this still works with /-/reload support.

(We guarantee that Run will only be called once per config, but NewIntegration may be called while the previous instance is still running)

[tpaschalis]

No, I don't think that there will be any issues.

I tested on a Linux VPS, changing various bits of configuration, enabling/disabling the integration and reloading each time; it seemed to work nicely.

[github-actions]

This PR has been automatically marked as stale because it has not had any activity in the past 30 days.
The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed in 7 days if there is no new activity.
Thank you for your contributions!

[tpaschalis]

Commenting here to un-stale. I want to check if any new ebpf packaging has been contributed upstream, especially after we changed to a new Debian base image.

[mattdurham]

Looks good to me