Yet Another Profiler

This project is in development phase.

This is yet another low-overhead kernel-assisted sampling-based CPU time continuous profile. It does not need instrumentation in the profiled binary.

A simple sampling eBPF program attached to a timer collects:

• stack traces
• sampled stack trace counts

The data collected from the kernel is analysed in user-space to summarise residency fraction. That is, for a stack trace, the percentage of samples that contained that path out of the total amount of samples.

This information exctracted from the collected data expresses, for a specific process, which functions are mostly executing.

How it works

The sampling eBPF probe is attached to a perf CPU clock software event.

The user and kernel stack traces that are running on the current CPU are available to the eBPF program that will run in the context of the interrupted process via the bpf_get_stackid eBPF helper. The user or kernel stack will be available depending on the context during which the process was interrupted.

The hard work of stack walking is made easy by the Linux kernel thanks to the fact that frame instruction pointers of the sampled stack traces are available in kernel space via the BPF_MAP_TYPE_STACK_TRACE eBPF map.

The information about how much a specific stack has been sampled is tracked with counters stored in an histogram eBPF map, which is keyed by:

• User stack ID
• Kernel stack ID
• PID to filter later on

and made available to userspace, alongside the stack traces.

In userspace symbolization is made with frame instruction pointer addresses read from the map.

Finally the information is extracted as percentage of profile time a stack trace has been executing.

Quickstart

$ sudo profiler --pid 591488
{"level":"debug","message":"loading ebpf object"}
{"level":"debug","message":"getting the loaded ebpf program"}
{"level":"debug","message":"opening the sampling software cpu block perf event"}
{"level":"debug","message":"attaching the ebpf program to the sampling perf event for cpu #0"}
{"level":"debug","message":"opening the sampling software cpu block perf event"}
{"level":"debug","message":"attaching the ebpf program to the sampling perf event for cpu #1"}
{"level":"debug","message":"opening the sampling software cpu block perf event"}
{"level":"debug","message":"attaching the ebpf program to the sampling perf event for cpu #2"}
{"level":"debug","message":"opening the sampling software cpu block perf event"}
{"level":"debug","message":"attaching the ebpf program to the sampling perf event for cpu #3"}
{"level":"debug","message":"opening the sampling software cpu block perf event"}
{"level":"debug","message":"attaching the ebpf program to the sampling perf event for cpu #4"}
{"level":"debug","message":"opening the sampling software cpu block perf event"}
{"level":"debug","message":"attaching the ebpf program to the sampling perf event for cpu #5"}
{"level":"debug","message":"opening the sampling software cpu block perf event"}
{"level":"debug","message":"attaching the ebpf program to the sampling perf event for cpu #6"}
{"level":"debug","message":"opening the sampling software cpu block perf event"}
{"level":"debug","message":"attaching the ebpf program to the sampling perf event for cpu #7"}
{"level":"info","message":"collecting data"}

^C{"level":"info","message":"terminating..."}
{"level":"info","message":"received signal, analysing data"}
{"level":"debug","message":"getting the stack traces ebpf map"}
{"level":"debug","message":"getting the stack trace counts (histogram) ebpf maps"}
{"level":"debug","message":"iterating over the retrieved histogram items"}

70% main(); compute(); matrix_multiply()
10% main(); read_data(); read_file()
 5% main(); compute(); matrix_multiply(); prepare()

Build

Build all

make profiler

eBPF probe only

make profiler/bpf

Thanks

• Pixie:
  • pixie-demos/ebpf-profiler
  • Building a continuous profiler
• Linux:
  • samples/bpf/trace_event_user.c
  • samples/bpf/trace_event_kern.c
• Brendan Gregg:
  • Linux eBPF Stack Trace Hack