To debug Ubuntu kernel with debug symbol package(dbgsym) is very easy. I have a very small note for it, but now I try to elaborate it in a blog with an example I just encountered.

  1. First of all, you need a kernel git tree, mine is located in ~/workspace/ubuntu-xenial

    • clone kernel source and checkout to the version we are going to use
      1
      2
      3
      git clone git://kernel.ubuntu.com/ubuntu/ubuntu-xenial ~/workspace/ubuntu-xenail
      cd ~/workspace/ubuntu-xenial
      git checkout git checkout Ubuntu-4.4.0-81.104

  2. Then, you need to install the kernel debug symbol package which matches the version you want to debug to. In this case, its 4.4.0-81

    • add ddebs source list
      1
      2
      3
      4
      5
         echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse
      deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
      deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
      sudo tee -a /etc/apt/sources.list.d/ddebs.list
    • add public key
      1
      sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01 C8CAB6595FDFF622
    • update/install dbgsym package
      1
      2
      sudo apt-get update
      sudo apt-get install linux-image-4.4.0-81-generic-dbgsym

  3. Here is our Kernel Oops

    BUG: unable to handle kernel NULL pointer dereference at (null)
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    [  553.073158] BUG: unable to handle kernel NULL pointer dereference at           (null)
    [  553.073179] IP: [<ffffffff8151f14d>] serial8250_do_startup+0x12d/0x650
    [  553.073195] PGD 0 
    [  553.073200] Oops: 0000 [#1] SMP 
    [  553.073208] Modules linked in: dell_uart_backlight(OE+) rfcomm drbg ansi_cprng ctr ccm nvram msr cmac bnep uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core v4l2_common videodev media arc4 btusb(OE) btrtl(OE) iwlmvm(OE) mac80211(OE) i2c_designware_platform i2c_designware_core intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp iwlwifi(OE) kvm_intel kvm dell_wmi_aio dell_wmi sparse_keymap cfg80211(OE) snd_hda_codec_hdmi(OE) snd_hda_codec_realtek(OE) snd_hda_codec_generic(OE) irqbypass hci_uart(OE) btqca(OE) btbcm(OE) snd_hda_intel(OE) snd_hda_codec(OE) btintel(OE) compat(OE) snd_hda_core(OE) rtsx_pci_ms memstick bluetooth snd_hwdep snd_pcm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_seq_midi snd_seq_midi_event aesni_intel aes_x86_64 lrw gf128mul glue_helper
    [  553.073380]  snd_rawmidi ablk_helper snd_seq cryptd snd_seq_device snd_timer snd dcdbas acpi_als shpchp serio_raw soundcore idma64 virt_dma kfifo_buf industrialio intel_lpss_acpi 8250_fintek intel_lpss_pci intel_lpss mac_hid acpi_pad parport_pc ppdev lp parport autofs4 btrfs xor raid6_pq hid_generic usbhid rtsx_pci_sdmmc i915_bpo intel_ips i2c_algo_bit psmouse drm_kms_helper e1000e(OE) ptp syscopyarea pps_core sysfillrect sysimgblt rtsx_pci fb_sys_fops drm ahci libahci wmi video i2c_hid hid fjes [last unloaded: dell_uart_backlight]
    [  553.073504] CPU: 3 PID: 5031 Comm: insmod Tainted: G           OE   4.4.0-81-generic #104-Ubuntu
    [  553.073521] Hardware name: Dell Inc. Unidentified System/      , BIOS 99.8.3[BCL_HID_DELL0501] 06/21/2017
    [  553.073539] task: ffff8800d67c8000 ti: ffff88011744c000 task.ti: ffff88011744c000
    [  553.073553] RIP: 0010:[<ffffffff8151f14d>]  [<ffffffff8151f14d>] serial8250_do_startup+0x12d/0x650
    [  553.073571] RSP: 0018:ffff88011744fac0  EFLAGS: 00010246
    [  553.073581] RAX: 0000000000000000 RBX: ffff8800d6fdbc08 RCX: 0000000000000000
    [  553.073594] RDX: 00000000000003f9 RSI: 0000000000000202 RDI: ffff8800d6fdbc08
    [  553.073607] RBP: ffff88011744fae8 R08: 0000000000000003 R09: ffffffff813fb08c
    [  553.073620] R10: ffffea00035e4400 R11: 0000000000000016 R12: 0000000000000000
    [  553.073633] R13: 0000000000000202 R14: 0000000000000000 R15: ffff8800d5993660
    [  553.073647] FS:  00007fb335f61700(0000) GS:ffff88011dd80000(0000) knlGS:0000000000000000
    [  553.073662] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    [  553.073673] CR2: 0000000000000000 CR3: 000000011833e000 CR4: 00000000003406e0
    [  553.073702] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    [  553.073723] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
    [  553.073736] Stack:
    [  553.073740]  ffff8800d6fdbc00 ffff8800d6fdbc08 ffff8800c35b3000 0000000000000029
    [  553.073758]  ffff8800d5993660 ffff88011744faf8 ffffffff8151f68b ffff88011744fb48
    [  553.073774]  ffffffffc0a2a7a3 ffffffff81290960 ffff8800c35b3268 ffff8800c35b3278
    [  553.073790] Call Trace:
    [  553.073797]  [<ffffffff8151f68b>] serial8250_startup+0x1b/0x20
    [  553.073812]  [<ffffffffc0a2a7a3>] dell_uart_bl_add+0xb3/0x333 [dell_uart_backlight]
    [  553.073828]  [<ffffffff81290960>] ? sysfs_do_create_link_sd.isra.2+0x70/0xb0
    [  553.073843]  [<ffffffff81491319>] acpi_device_probe+0x49/0x100
    [  553.073856]  [<ffffffff81562ee2>] driver_probe_device+0x222/0x4a0
    [  553.073868]  [<ffffffff815631e4>] __driver_attach+0x84/0x90
    [  553.073880]  [<ffffffff81563160>] ? driver_probe_device+0x4a0/0x4a0
    [  553.073892]  [<ffffffff81560b0c>] bus_for_each_dev+0x6c/0xc0
    [  553.073904]  [<ffffffff8156269e>] driver_attach+0x1e/0x20
    [  553.073915]  [<ffffffff815621db>] bus_add_driver+0x1eb/0x280
    [  553.073926]  [<ffffffffc0008000>] ? 0xffffffffc0008000
    [  553.073937]  [<ffffffff81563af0>] driver_register+0x60/0xe0
    [  553.073948]  [<ffffffff8149126c>] acpi_bus_register_driver+0x3c/0x50
    [  553.073961]  [<ffffffffc0008010>] dell_uart_bl_init+0x10/0x1000 [dell_uart_backlight]
    [  553.073977]  [<ffffffff81002123>] do_one_initcall+0xb3/0x200
    [  553.073990]  [<ffffffff811ed723>] ? kmem_cache_alloc_trace+0x183/0x1f0
    [  553.074006]  [<ffffffff8118dd23>] do_init_module+0x5f/0x1cf
    [  553.074018]  [<ffffffff8110aaaf>] load_module+0x166f/0x1c10
    [  553.074029]  [<ffffffff81107050>] ? __symbol_put+0x60/0x60
    [  553.074041]  [<ffffffff812153b0>] ? kernel_read+0x50/0x80
    [  553.074052]  [<ffffffff8110b294>] SYSC_finit_module+0xb4/0xe0
    [  553.074064]  [<ffffffff8110b2de>] SyS_finit_module+0xe/0x10
    [  553.074076]  [<ffffffff81840a72>] entry_SYSCALL_64_fastpath+0x16/0x71
    [  553.074088] Code: 00 00 40 0f 84 d2 00 00 00 83 bb fc 00 00 00 0c 0f 84 b7 03 00 00 8b 4b 78 85 c9 0f 85 ed 01 00 00 48 8b 83 f8 01 00 00 48 89 df <ff> 10 85 c0 41 89 c4 0f 84 df 00 00 00 66 83 bb d0 01 00 00 00 
    [  553.074166] RIP  [<ffffffff8151f14d>] serial8250_do_startup+0x12d/0x650
    [  553.074180]  RSP <ffff88011744fac0>
    [  553.074187] CR2: 0000000000000000
    [  553.078892] ---[ end trace 3d07b7615015991a ]---

  4. Before starting to look deeper into the log, I would like to setup gdb environment

    startup gdb
    1
    2
    3
    gdb /usr/lib/debug/boot/vmlinux-4.4.0-81-generic
    or
    gdb /usr/lib/debug/boot/vmlinux-`uname -r`
    find the build path and replace it with current git tree
    find the build path
    1
    2
    gdb$ list printk
    1887    /build/linux-cs3yMe/linux-4.4.0/kernel/printk/printk.c: No such file or directory.
    The path, /build/linux-cs3yMe, does not match the kernel directory in our computer.
    replace it with current git tree
    1
    gdb$ set substitute-path /build/linux-cs3yMe/linux-4.4.0 /home/acelan/workspace/ubuntu-xenial
    Now, list command should work 
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    gdb$ list printk
    1887     * printf(3)
    1888     *
    1889     * See the vsnprintf() documentation for format string extensions over C99.
    1890     */
    1891    asmlinkage __visible int printk(const char *fmt, ...)
    1892    {
    1893            printk_func_t vprintk_func;
    1894            va_list args;
    1895            int r;
    1896

  5. We can check what went wrong in the function serial8250_do_startup

    IP
    1
    IP: [<ffffffff8151f14d>] serial8250_do_startup+0x12d/0x650
    The format of this string is [function name]+[error offset]/[total assembly code length] We only care about where it went wrong, so we can pass the function name + offset to list the code around offset
    IP: [] serial8250_do_startup+0x12d/0x650
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    gdb$ list *(serial8250_do_startup+0x12d)
    0xffffffff8151f14d is in serial8250_do_startup (/build/linux-cs3yMe/linux-4.4.0/drivers/tty/serial/8250/8250_port.c:1942).
    1937                        up->port.flags & UPF_BUG_THRE) {
    1938                            up->bugs |= UART_BUG_THRE;
    1939                    }
    1940            }
    1941
    1942            retval = up->ops->setup_irq(up);
    1943            if (retval)
    1944                    goto out;
    1945
    1946            /*
    The oops is from up->ops->setup_irq(), and from the error message, I believe we didn't setup "setup_irq" function pointer, so it leads to the NULL pointer dereference error.

Reference

  1. Debug Symbol Packages
  2. Kernel Debugging Tricks