Linux Kernel Debugging

Create a kernel to be run in QEMU and debugged with gdb

The following steps try to lay out the steps necessary to compile a kernel for being debugged in QEMU.

mkdir -p linux-src
curl $( curl -s https://www.kernel.org/releases.json | jq -r '.releases[] | select(.moniker == "stable") | .source' ) | tar xJ -C linux-src --strip-components=1

Create the .config file with a default configuration:

cd linux-src
make defconfig

Enable the config options

CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT (which relies on the toolchain's implicit default DWARF version) and
GDB_SCRIPTS (which causes the Makefile to call make the directory /scripts/gdb/ and create a symbolic link from vmlinux-gdb.py to scripts/gdb/vmlinux-gdb.py.

(I am not sure if the latter is really required, but I've seen it recommended):

./scripts/config --enable DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT 
./scripts/config --enable GDB_SCRIPTS

After changing config options with ./scripts/config, config options that are dependent on the changed ones need to be updated (at least this what I think the following make target does):

make olddefconfig

TODO: the previous command displayed .config:4980:warning: override: DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT changes choice state.

We're now ready to build the Kernel. The -j option is used to assign the number of cpus (here: 2 less than available):

make -j $(( $(nproc) - 2 ))

Since we're at it, we also make the gdb scripts:

make scripts_gdb

Setting breakpoints in the kernel

The following is a simple demonstration on using gdb to set breakpoints and stepping through the kernel.

First, the newly built kernel is started in QEMU:

qemu-system-x86_64 -kernel arch/x86_64/boot/bzImage -s -S -append kaslr

The command line option -s is shorthand for -gdb tcp::1234 which in turn specifies the port to which gdb can connect to to debug the kernel.

The command line option -S starts the kernel in a suspended (frozen) mode and waits for the debugger to connect to it.

We now connect to the debugee with gdb:

gdb vmlinux

Connect to the debugee:

(gdb) target remote :1234
Remote debugging using :1234
0x000000000000fff0 in exception_stacks ()

A (hardware) breakpoint (hbreak) is set at an early stage in the execution of the kernel.

When I tried to find the earliest breakable line in Linux version 6.9.8, I found this to be line 208 of arch/x86/kernel/head_64.S.

(gdb) hbreak arch/x86/kernel/head_64.S:208
Hardware assisted breakpoint 1 at 0xffffffff8102de38: file arch/x86/kernel/head_64.S, line 208.

Run the kernel until it hits the breakpoint:

(gdb) cont
Continuing.

Breakpoint 1, secondary_startup_64 () at arch/x86/kernel/head_64.S:208
208		movl	$(X86_CR4_PAE | X86_CR4_LA57), %edx

Show the backtrace:

(gdb) bt
#0  secondary_startup_64 () at arch/x86/kernel/head_64.S:208
#1  0x0000000000000000 in ?? ()

Delete the breakpoint, add another one and continue

(gdb) delete 1lk
(gdb) hbreak arch/x86/kernel/head_64.S:420
Hardware assisted breakpoint 2 at 0xffffffff8102df5e: file arch/x86/kernel/head_64.S, line 420.
(gdb) cont
Continuing.

Breakpoint 2, secondary_startup_64 () at arch/x86/kernel/head_64.S:420
420		callq	*initial_code(%rip)

This callq instruction jumps to x86_64_start_kernel

(gdb) delete 2
(gdb) s
x86_64_start_kernel (real_mode_data=0x14750 <entry_stack_storage+1872> <error: Cannot access memory at address 0x14750>) at arch/x86/kernel/head64.c:426
…
(gdb) bt
#0  x86_64_start_kernel (real_mode_data=0x14750 <entry_stack_storage+1872> <error: Cannot access memory at address 0x14750>) at arch/x86/kernel/head64.c:426
#1  0xffffffff8102df64 in secondary_startup_64 () at arch/x86/kernel/head_64.S:420
#2  0x0000000000000000 in ?? ()

Setting breakpoints on a syscall

In order to set a breakpoint on a syscall, we need a process that executes the syscall because for obvious reasons the syscall cannot be called from within the kernel.

Therefore, we create an init executable which is the default first process started by the kernel. This init will then call write twice and shutdown the kernel. This gives us the possiblity to break on the write syscall (we actually ksys_write).

init.S

This assembler code is the source for the init process:

.section .data
    text_1:      .ascii "\033[91m" "first line"  "\033[0m\n"
    text_1_len = . - text_1
    text_2:      .ascii "\033[92m" "second line" "\033[0m\n"
    text_2_len = . - text_2

.section .text
    .globl _start

_start:

 # Call write syscall (rax = 1) which takes the three parameters
 #    rdi = filedescriptor
 #    rsi = pointer to message
 #    rdx = length of message

 #
 # Line 1
 #
   mov $1          , %rax
   mov $1          , %rdi
   lea text_1(%rip), %rsi
   mov $text_1_len , %rdx
   syscall

 #
 # Line 2
 #
   mov $1          , %rax
   mov $1          , %rdi
   lea text_2(%rip), %rsi
   mov $text_2_len , %rdx
   syscall

 #
 # call reboot syscall with shutdown
 #
   mov $169        , %rax # sys_reboot
   mov $0xfee1dead , %rdi # magic number 1
   mov $0x28121969 , %rsi # magic number 2
   mov $0x4321fedc , %rdx # magic number 3
   mov $0x00000000 , %r10 # LINUX_REBOOT_CMD_POWER_OFF
   syscall

It is compiled like so:

as --64 -o init.o init.S
ld      -o init   init.o

Creating an init ram disk

In order for the kernel to access the init executable, it needs to be put into an init ram disk:

echo init | cpio -H newc -o > init.cpio

Starting the kernel in QEMU

With the cpio archive, we can now start the kernel in QEMU in the directory where linux was compiled (note the -initrd option):

qemu-system-x86_64 -kernel arch/x86_64/boot/bzImage -s -S -append nokaslr -initrd ../init.cpio

Setting the breakpoint

$ gdb vmlinux
(gdb) break ksys_write
…
(gdb) cont
…
(gdb) step
…
(gdb) finish
…
(gdb) cont