To debug Ubuntu kernel with debug symbol package(dbgsym) is very easy. I have a very small note for it, but now I try to elaborate it in a blog with an example I just encountered.

  1. First of all, you need a kernel git tree, mine is located in ~/workspace/ubuntu-xenial

    • clone kernel source and checkout to the version we are going to use
      1
      2
      3
      git clone git://kernel.ubuntu.com/ubuntu/ubuntu-xenial ~/workspace/ubuntu-xenail
      cd ~/workspace/ubuntu-xenial
      git checkout git checkout Ubuntu-4.4.0-81.104
  2. Then, you need to install the kernel debug symbol package which matches the version you want to debug to. In this case, its 4.4.0-81

    • add ddebs source list
      1
      2
      3
      4
      echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse
      deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
      deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \

      sudo tee -a /etc/apt/sources.list.d/ddebs.list
    • add public key
      1
      sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01 C8CAB6595FDFF622
    • update/install dbgsym package
      1
      2
      sudo apt-get update
      sudo apt-get install linux-image-4.4.0-81-generic-dbgsym
  3. Here is our Kernel Oops

    BUG: unable to handle kernel NULL pointer dereference at (null)
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    [  553.073158] BUG: unable to handle kernel NULL pointer dereference at           (null)
    [ 553.073179] IP: [<ffffffff8151f14d>] serial8250_do_startup+0x12d/0x650
    [ 553.073195] PGD 0
    [ 553.073200] Oops: 0000 [#1] SMP
    [ 553.073208] Modules linked in: dell_uart_backlight(OE+) rfcomm drbg ansi_cprng ctr ccm nvram msr cmac bnep uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core v4l2_common videodev media arc4 btusb(OE) btrtl(OE) iwlmvm(OE) mac80211(OE) i2c_designware_platform i2c_designware_core intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp iwlwifi(OE) kvm_intel kvm dell_wmi_aio dell_wmi sparse_keymap cfg80211(OE) snd_hda_codec_hdmi(OE) snd_hda_codec_realtek(OE) snd_hda_codec_generic(OE) irqbypass hci_uart(OE) btqca(OE) btbcm(OE) snd_hda_intel(OE) snd_hda_codec(OE) btintel(OE) compat(OE) snd_hda_core(OE) rtsx_pci_ms memstick bluetooth snd_hwdep snd_pcm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_seq_midi snd_seq_midi_event aesni_intel aes_x86_64 lrw gf128mul glue_helper
    [ 553.073380] snd_rawmidi ablk_helper snd_seq cryptd snd_seq_device snd_timer snd dcdbas acpi_als shpchp serio_raw soundcore idma64 virt_dma kfifo_buf industrialio intel_lpss_acpi 8250_fintek intel_lpss_pci intel_lpss mac_hid acpi_pad parport_pc ppdev lp parport autofs4 btrfs xor raid6_pq hid_generic usbhid rtsx_pci_sdmmc i915_bpo intel_ips i2c_algo_bit psmouse drm_kms_helper e1000e(OE) ptp syscopyarea pps_core sysfillrect sysimgblt rtsx_pci fb_sys_fops drm ahci libahci wmi video i2c_hid hid fjes [last unloaded: dell_uart_backlight]
    [ 553.073504] CPU: 3 PID: 5031 Comm: insmod Tainted: G OE 4.4.0-81-generic #104-Ubuntu
    [ 553.073521] Hardware name: Dell Inc. Unidentified System/ , BIOS 99.8.3[BCL_HID_DELL0501] 06/21/2017
    [ 553.073539] task: ffff8800d67c8000 ti: ffff88011744c000 task.ti: ffff88011744c000
    [ 553.073553] RIP: 0010:[<ffffffff8151f14d>] [<ffffffff8151f14d>] serial8250_do_startup+0x12d/0x650
    [ 553.073571] RSP: 0018:ffff88011744fac0 EFLAGS: 00010246
    [ 553.073581] RAX: 0000000000000000 RBX: ffff8800d6fdbc08 RCX: 0000000000000000
    [ 553.073594] RDX: 00000000000003f9 RSI: 0000000000000202 RDI: ffff8800d6fdbc08
    [ 553.073607] RBP: ffff88011744fae8 R08: 0000000000000003 R09: ffffffff813fb08c
    [ 553.073620] R10: ffffea00035e4400 R11: 0000000000000016 R12: 0000000000000000
    [ 553.073633] R13: 0000000000000202 R14: 0000000000000000 R15: ffff8800d5993660
    [ 553.073647] FS: 00007fb335f61700(0000) GS:ffff88011dd80000(0000) knlGS:0000000000000000
    [ 553.073662] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    [ 553.073673] CR2: 0000000000000000 CR3: 000000011833e000 CR4: 00000000003406e0
    [ 553.073702] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    [ 553.073723] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
    [ 553.073736] Stack:
    [ 553.073740] ffff8800d6fdbc00 ffff8800d6fdbc08 ffff8800c35b3000 0000000000000029
    [ 553.073758] ffff8800d5993660 ffff88011744faf8 ffffffff8151f68b ffff88011744fb48
    [ 553.073774] ffffffffc0a2a7a3 ffffffff81290960 ffff8800c35b3268 ffff8800c35b3278
    [ 553.073790] Call Trace:
    [ 553.073797] [<ffffffff8151f68b>] serial8250_startup+0x1b/0x20
    [ 553.073812] [<ffffffffc0a2a7a3>] dell_uart_bl_add+0xb3/0x333 [dell_uart_backlight]
    [ 553.073828] [<ffffffff81290960>] ? sysfs_do_create_link_sd.isra.2+0x70/0xb0
    [ 553.073843] [<ffffffff81491319>] acpi_device_probe+0x49/0x100
    [ 553.073856] [<ffffffff81562ee2>] driver_probe_device+0x222/0x4a0
    [ 553.073868] [<ffffffff815631e4>] __driver_attach+0x84/0x90
    [ 553.073880] [<ffffffff81563160>] ? driver_probe_device+0x4a0/0x4a0
    [ 553.073892] [<ffffffff81560b0c>] bus_for_each_dev+0x6c/0xc0
    [ 553.073904] [<ffffffff8156269e>] driver_attach+0x1e/0x20
    [ 553.073915] [<ffffffff815621db>] bus_add_driver+0x1eb/0x280
    [ 553.073926] [<ffffffffc0008000>] ? 0xffffffffc0008000
    [ 553.073937] [<ffffffff81563af0>] driver_register+0x60/0xe0
    [ 553.073948] [<ffffffff8149126c>] acpi_bus_register_driver+0x3c/0x50
    [ 553.073961] [<ffffffffc0008010>] dell_uart_bl_init+0x10/0x1000 [dell_uart_backlight]
    [ 553.073977] [<ffffffff81002123>] do_one_initcall+0xb3/0x200
    [ 553.073990] [<ffffffff811ed723>] ? kmem_cache_alloc_trace+0x183/0x1f0
    [ 553.074006] [<ffffffff8118dd23>] do_init_module+0x5f/0x1cf
    [ 553.074018] [<ffffffff8110aaaf>] load_module+0x166f/0x1c10
    [ 553.074029] [<ffffffff81107050>] ? __symbol_put+0x60/0x60
    [ 553.074041] [<ffffffff812153b0>] ? kernel_read+0x50/0x80
    [ 553.074052] [<ffffffff8110b294>] SYSC_finit_module+0xb4/0xe0
    [ 553.074064] [<ffffffff8110b2de>] SyS_finit_module+0xe/0x10
    [ 553.074076] [<ffffffff81840a72>] entry_SYSCALL_64_fastpath+0x16/0x71
    [ 553.074088] Code: 00 00 40 0f 84 d2 00 00 00 83 bb fc 00 00 00 0c 0f 84 b7 03 00 00 8b 4b 78 85 c9 0f 85 ed 01 00 00 48 8b 83 f8 01 00 00 48 89 df <ff> 10 85 c0 41 89 c4 0f 84 df 00 00 00 66 83 bb d0 01 00 00 00
    [ 553.074166] RIP [<ffffffff8151f14d>] serial8250_do_startup+0x12d/0x650
    [ 553.074180] RSP <ffff88011744fac0>
    [ 553.074187] CR2: 0000000000000000
    [ 553.078892] ---[ end trace 3d07b7615015991a ]---

  4. Before starting to look deeper into the log, I would like to setup gdb environment

    startup gdb
    1
    2
    3
    gdb /usr/lib/debug/boot/vmlinux-4.4.0-81-generic
    or
    gdb /usr/lib/debug/boot/vmlinux-`uname -r`
    find the build path and replace it with current git tree
    find the build path
    1
    2
    gdb$ list printk
    1887 /build/linux-cs3yMe/linux-4.4.0/kernel/printk/printk.c: No such file or directory.
    The path, /build/linux-cs3yMe, does not match the kernel directory in our computer.
    replace it with current git tree
    1
    gdb$ set substitute-path /build/linux-cs3yMe/linux-4.4.0 /home/acelan/workspace/ubuntu-xenial
    Now, list command should work
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    gdb$ list printk
    1887 * printf(3)
    1888 *
    1889 * See the vsnprintf() documentation for format string extensions over C99.
    1890 */
    1891 asmlinkage __visible int printk(const char *fmt, ...)
    1892 {

    1893 printk_func_t vprintk_func;
    1894 va_list args;
    1895 int r;
    1896

  5. We can check what went wrong in the function serial8250_do_startup

    IP
    1
    IP: [<ffffffff8151f14d>] serial8250_do_startup+0x12d/0x650
    The format of this string is [function name]+[error offset]/[total assembly code length] We only care about where it went wrong, so we can pass the function name + offset to list the code around offset
    IP: [] serial8250_do_startup+0x12d/0x650
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    gdb$ list *(serial8250_do_startup+0x12d)
    0xffffffff8151f14d is in serial8250_do_startup (/build/linux-cs3yMe/linux-4.4.0/drivers/tty/serial/8250/8250_port.c:1942).
    1937 up->port.flags & UPF_BUG_THRE) {
    1938 up->bugs |= UART_BUG_THRE;
    1939 }
    1940 }
    1941
    1942 retval = up->ops->setup_irq(up);
    1943 if (retval)
    1944 goto out;
    1945
    1946 /*
    The oops is from up->ops->setup_irq(), and from the error message, I believe we didn't setup "setup_irq" function pointer, so it leads to the NULL pointer dereference error.

Reference

  1. Debug Symbol Packages
  2. Kernel Debugging Tricks