跟風來學一下最新的技術 eBPF(extended Berkeley Packet Filter),本來想看看對 debug kernel 有沒有什麼幫助,仔細研究了一下發現 eBPF 更多是在觀察跟統計 kernel 的行為上,對 debug kernel 的幫助不大。不過時間都花了,就稍微記綠一下,主要還是著重在對 debug 有幫助的東西上。

  1. 簡介 eBPF 能做什麼 目前 eBPF 支援以下幾種類型的程式,但是這邊只研究 kprobe 跟 tracepoint。

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    bool is_socket = strncmp(event, "socket", 6) == 0;		// a network packet filter
    bool is_kprobe = strncmp(event, "kprobe/", 7) == 0; // determine whether a kprobe should fire or not
    bool is_kretprobe = strncmp(event, "kretprobe/", 10) == 0; // determine whether a kretprobe should fire or not
    bool is_tracepoint = strncmp(event, "tracepoint/", 11) == 0; // determine whether a tracepoint should fire or not
    bool is_xdp = strncmp(event, "xdp", 3) == 0; // a network packet filter run from the device-driver receive path
    bool is_perf_event = strncmp(event, "perf_event", 10) == 0; // determine whether a perf event handler should fire or not
    bool is_cgroup_skb = strncmp(event, "cgroup/skb", 10) == 0; // a network packet filter for control groups
    bool is_cgroup_sk = strncmp(event, "cgroup/sock", 11) == 0; // a network packet filter for control groups that is allowed to modify socket options
    bool is_sockops = strncmp(event, "sockops", 7) == 0; // a program for setting socket parameters
    bool is_sk_skb = strncmp(event, "sk_skb", 6) == 0; // a network packet filter for forwarding packets between sockets
    userspace app 與 kernel module 溝通的資料結構有下列幾種
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    BPF_MAP_TYPE_HASH:		a hash table
    BPF_MAP_TYPE_ARRAY: an array map, optimized for fast lookup speeds, often used for counters
    BPF_MAP_TYPE_PROG_ARRAY: an array of file descriptors corresponding to eBPF programs; used to implement jump tables and sub-programs to handle specific packet protocols
    BPF_MAP_TYPE_PERCPU_ARRAY: a per-CPU array, used to implement histograms of latency
    BPF_MAP_TYPE_PERF_EVENT_ARRAY: stores pointers to struct perf_event, used to read and store perf event counters
    BPF_MAP_TYPE_CGROUP_ARRAY: stores pointers to control groups
    BPF_MAP_TYPE_PERCPU_HASH: a per-CPU hash table
    BPF_MAP_TYPE_LRU_HASH: a hash table that only retains the most recently used items
    BPF_MAP_TYPE_LRU_PERCPU_HASH: a per-CPU hash table that only retains the most recently used items
    BPF_MAP_TYPE_LPM_TRIE: a longest-prefix match trie, good for matching IP addresses to a range
    BPF_MAP_TYPE_STACK_TRACE: stores stack traces
    BPF_MAP_TYPE_ARRAY_OF_MAPS: a map-in-map data structure
    BPF_MAP_TYPE_HASH_OF_MAPS: a map-in-map data structure
    BPF_MAP_TYPE_DEVICE_MAP: for storing and looking up network device references
    BPF_MAP_TYPE_SOCKET_MAP: stores and looks up sockets and allows socket redirection with BPF helper functions

  2. 為什麼要用 eBPF

  • Q. kprobe module 直接寫就好了,為什麼要使用 eBPF 包起來寫?
  • A. eBPF code runs in vm, never panic the running kernel
  1. 實用工具 bcc, 可以在下面 reference 的 BPF Compiler Collection (BCC) 找到安裝方法,或是直接裝 snap package
    1
    2
    snap install bcc
    # and the tools are prefixed by bcc, e.g. sudo bcc.biotop
    1
    2
    3
    4
    5
    /*
    * This program traces functions and frequency counts them with their entire
    * stack trace, summarized in-kernel for efficiency.
    */
    sudo /usr/share/bcc/tools/stackcount -K hrtimer_init_sleeper
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    /* 
    * trace probes functions you specify and displays trace messages if a particular
    * condition is met. You can control the message format to display function
    * arguments and return values.
    */
    /*
    * "retval": "PT_REGS_RC(ctx)",
    * "arg1": "PT_REGS_PARM1(ctx)",
    * "arg2": "PT_REGS_PARM2(ctx)",
    * "arg3": "PT_REGS_PARM3(ctx)",
    * "arg4": "PT_REGS_PARM4(ctx)",
    * "arg5": "PT_REGS_PARM5(ctx)",
    * "arg6": "PT_REGS_PARM6(ctx)",
    * "$uid": "(unsigned)(bpf_get_current_uid_gid() & 0xffffffff)",
    * "$gid": "(unsigned)(bpf_get_current_uid_gid() >> 32)",
    * "$pid": "(unsigned)(bpf_get_current_pid_tgid() & 0xffffffff)",
    * "$tgid": "(unsigned)(bpf_get_current_pid_tgid() >> 32)",
    * "$cpu": "bpf_get_smp_processor_id()"
    */
    sudo /usr/share/bcc/tools/trace '::sys_execve "%s", arg1'
    PID COMM FUNC -
    4402 bash sys_execve /usr/bin/man
    4411 man sys_execve /usr/local/bin/less
    4411 man sys_execve /usr/bin/less
    4410 man sys_execve /usr/local/bin/nroff
    4410 man sys_execve /usr/bin/nroff
    4409 man sys_execve /usr/local/bin/tbl
    4409 man sys_execve /usr/bin/tbl
    4408 man sys_execve /usr/local/bin/preconv
    4408 man sys_execve /usr/bin/preconv
    4415 nroff sys_execve /usr/bin/locale
    4416 nroff sys_execve /usr/bin/groff
    4418 groff sys_execve /usr/bin/grotty
    4417 groff sys_execve /usr/bin/troff
    ^C
    1
    2
    3
    4
    5
    6
    sudo /usr/share/bcc/tools/trace 't:block:block_rq_complete "sectors=%d", args->nr_sector' -T
    TIME PID COMM FUNC -
    01:23:51 0 swapper/0 block_rq_complete sectors=8
    01:23:55 10017 kworker/u64: block_rq_complete sectors=1
    01:23:55 0 swapper/0 block_rq_complete sectors=8
    ^C
    1
    2
    sudo /usr/share/bcc/tools/trace 'r::__kmalloc (retval == 0) "kmalloc failed!"'
    Trace returns from __kmalloc which returned a null pointer
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    /*
    * This program traces functions, tracepoints, or USDT probes that match a
    * specified pattern, and when Ctrl-C is hit prints a summary of their count
    * while tracing.
    */
    sudo /usr/share/bcc/tools/funccount 'vfs_*'
    Tracing... Ctrl-C to end.
    ^C
    FUNC COUNT
    vfs_create 1
    vfs_rename 1
    vfs_fsync_range 2
    vfs_lock_file 30
    vfs_fstatat 152
    vfs_fstat 154
    vfs_write 166
    vfs_getattr_nosec 262
    vfs_getattr 262
    vfs_open 264
    vfs_read 470
    Detaching...
    1
    2
    3
    4
    5
    6
    7
    8
    9
    sudo /usr/share/bcc/tools/funccount t:block:*
    Tracing 19 functions for "t:block:*"... Hit Ctrl-C to end.
    ^C
    FUNC COUNT
    block:block_rq_complete 7
    block:block_rq_issue 7
    block:block_getrq 7
    block:block_rq_insert 7
    Detaching...
    1
    2
    3
    4
    5
    6
    7
    /*
    * This program traces hard interrupts (irqs), and stores timing statistics
    * in-kernel for efficiency.
    */
    sudo /usr/share/bcc/tools/hardirqs
    # -d : distribution histogram
    sudo /usr/share/bcc/tools/hardirqs -d
    1
    2
    3
    4
    5
    6
    7
    /*
    * This program traces soft interrupts (irqs), and stores timing statistics
    * in-kernel for efficiency.
    */
    sudo /usr/share/bcc/tools/softirqs
    # -d : distribution histogram
    sudo /usr/share/bcc/tools/softirqs -d

參考文件

  1. eBPF 簡史 - 非常全面的中文 eBPF 介紹。 Linux Enhanced BPF (eBPF) Tracing Tools - eBPF 的大本營。所以 eBPF 的學習資源都可以在這找到。 BPF Compiler Collection (BCC) - 使用 python 寫出的工具集,完美的將 kernel module 及 userspace app 整合到 python 裡,大幅降低 eBPF 的使用難度。 bcc Tutorial - 詳細的 BCC 工具程式說明 bcc Reference Guide - BCC API 文件 A dynamic tracer for Linux - 比 BCC 更精簡的一行式程式,目前可以使用 kprobe/kretprobe 等 kernel 提供的除錯函數。 BPF samples in Kernel - Linux Kernel 收集的一些 BPF 範例。