Fork us on GitHub Follow us on Facebook Follow us on Twitter

Changes between Version 4 and Version 5 of DebuggingWithQEMUAndGDB


Ignore:
Timestamp:
2020-01-14T15:36:41Z (3 months ago)
Author:
Martin Decky
Comment:

refresh the text a bit

Legend:

Unmodified
Added
Removed
Modified
  • DebuggingWithQEMUAndGDB

    v4 v5  
    66}}}
    77
    8 The combination of QEMU and GDB allows HelenOS to be comfortably debugged either on the assembly or the source code level. For detailed information on low-level debugging, see for example this [http://d3s.mff.cuni.cz/teaching/crash_dump_analysis/ course] on crash dump analysis.
     8The combination of QEMU and GDB allows HelenOS to be comfortably debugged either on the assembly or the source code level.
    99
    1010== Preparing the build ==
    1111
    12 The default HelenOS build should produce unstripped binaries. If necessary, this can be enforced by making sure the `Strip binaries` build configuration option is not checked. Unstripped binaries come with symbols, but do not contain any fancier debugging information. In order to get maximum out of the debug build, make sure to configure HelenOS with the `Line debugging information` option. Another thing which may impede debugging is optimization, so consider changing optimization levels to 0 using the `OPTIMIZATION` variable in the respective Makefiles ([browser:mainline/kernel/Makefile] or [browser:mainline/uspace/Makefile.common]). When everything is set, make sure to rebuild with the new settings.
     12The default HelenOS build should produce unstripped binaries. If necessary, this can be enforced by making sure the `Strip binaries` build configuration option is not checked. Unstripped binaries come with symbols, but do not contain any fancier debugging information. In order to get maximum out of the debug build, make sure to configure HelenOS with the `Line debugging information` option. Another thing which may impede debugging is optimization, so consider changing optimization levels to 0 using the `Optimization level` configuration option. When everything is set, make sure to rebuild with the new settings.
    1313
    1414== Starting QEMU ==
    1515
    16 QEMU provides two command line options for debugging with GDB: `-s` and `-S`. The former instructs QEMU to listen for GDB connections on localhost:1234 (but does not wait for it) and the latter stops the guest CPU at startup so that debugging is possible from the very beginning. When starting emulation using the [browser:mainline/tools/ew.py] script, one can add these options like this:
     16QEMU provides two command line options for debugging with GDB: `-s` and `-S`. The former instructs QEMU to listen for GDB connections on localhost:1234 (but does not wait for it) and the latter stops the guest CPU at startup so that debugging is possible from the very beginning. When starting emulation using the `tools/ew.py` script, one can add these options like this:
    1717
    1818{{{
     
    2222== Connecting GDB to QEMU ==
    2323
    24 Once QEMU is started with the `-s` (and optionally also the `-S`) option, it is possible to connect GDB to it. For our purposes, we will assume the respective cross-GDB built by the [browser:mainline/tools/toolchain.sh] is ready to be used and named as `/usr/local/cross/ia32/bin/i686-pc-linux-gnu-gdb`. Needless to say, the cross-GDB should always match the architecture of the HelenOS guest.
     24Once QEMU is started with the `-s` (and optionally also the `-S`) option, it is possible to connect GDB to it. For our purposes, we will assume the respective cross-GDB built by the `tools/toolchain.sh` is ready to be used and named as `/usr/local/cross/bin/i686-helenos-gdb`. Needless to say, the cross-GDB should always match the architecture of the HelenOS guest.
    2525
    2626{{{
    27 $ /usr/local/cross/ia32/bin/i686-pc-linux-gnu-gdb
     27$ /usr/local/cross/bin/i686-helenos-gdb
    2828GNU gdb (GDB) 7.11
    2929Copyright (C) 2016 Free Software Foundation, Inc.
     
    5151}}}
    5252
    53 Note that if QEMU was not started with the `-S` option, you will have to manually break into the debugger by pressing `Ctrl-C`.
     53Note that if QEMU was not started with the `-S` option, you will have to manually break into the debugger by pressing `Ctrl+C`.
    5454
    5555== Loading symbols ==
    5656
    57 Depending on what is the subject of our debugging session, we will need some symbols. The easiest case is kernel debugging. In that case, we simply load the kernel symbols from `kernel.raw`:
     57Depending on what is the subject of our debugging session, we will need some symbols. The easiest case is kernel debugging. In that case, we simply load the kernel symbols from `kernel.elf`:
    5858
    5959{{{
    60 (gdb) symbol-file kernel/kernel.raw
    61 Reading symbols from kernel/kernel.raw...done.
     60(gdb) symbol-file dist/boot/kernel.elf
     61Reading symbols from dist/boot/kernel.elf...done.
    6262}}}
    6363
    64 Debugging userspace tasks is a little more complicated because unlike the kernel, there will be multiple user processes running at the same time, so setting a mere breakpoint on a user address will probably not do. We will need to get some assistance from the kernel.
     64Debugging user space tasks is a little more complicated because unlike the kernel, there will be multiple user processes running at the same time, so setting a mere breakpoint on a user address will probably not do. We will need to get some assistance from the kernel.
    6565
    6666== Setting and hitting breakpoints ==
    6767
    68 To set a breakpoint, for example on the kernel function that handles invalid memory accesses from userspace, we type the following command:
     68To set a breakpoint, for example on the kernel function that handles invalid memory accesses from user space, we type the following command:
    6969
    7070{{{
     
    7474
    7575{{{#!box type=warning
    76 Sadly the set breakpoint is not always hit. At this point, the cause of this behavior is unknown.
     76Note that the default breakpoint mechanism modifies the code running in QEMU by inserting a breakpoint instruction (`INT3` on ia32). Thus if you setup a breakpoint using the `break` command before the kernel boots or an user space task loads, the breakpoint instruction might get overwritten and the breakpoint rendered ineffective. To avoid this situation, you can use hardware-assisted breakpoints using the `hbreak` command. Unfortunately, the number of hardware-assisted breakpoints is limited.
    7777}}}
    7878
     
    9898}}}
    9999
    100 Note that at any time while the guest is running, you can break into the debugger also by pressing the `Ctrl-C` combo.
     100Note that at any time while the guest is running, you can break into the debugger also by pressing the `Ctrl+C` combo.
    101101
    102102Now that a breakpoint in the kernel was hit, we can inspect the state of the kernel a little bit. Typing `bt` will show us the stack trace of the current kernel thread:
     
    127127== Useful macros ==
    128128
    129 When debugging the kernel, it is sometimes useful to find out some information about the current process. For that, we will need to mimic the computation of the address of the `THE` structure. We will start by defining a macro:
     129When debugging the kernel, it is sometimes useful to find out some information about the current task. For that, we will need to mimic the computation of the address of the `CURRENT` structure. We will start by defining a macro:
    130130
    131131{{{
    132 (gdb) macro define the ((the_t *) ((uintptr_t )$esp & ~0x1fff))
     132(gdb) macro define CURRENT ((current_t *) ((uintptr_t) $esp & ~0x1fff))
    133133}}}
    134134
     
    136136
    137137{{{
    138 (gdb) p the->task->name
    139 $3 = "tester\000it\000\000\000\000\000\000\000\000\000\000"
     138(gdb) x/s CURRENT->task->name
     1390x879748c8:     "/app/tester"
    140140}}}
    141141
     
    144144== Switching to the user context ==
    145145
    146 Let us assume that we have somehow found the values of the userspace registers EIP, ESP and EBP (for example by inspecting the `istate` structure as shown above) and know the name of the process (for example `tester` from the example above). Before we can add symbol information for this process, we need to find out the load address of its `.text` section (for some reason, GDB does not use the information provided in the ELF file):
     146Let us assume that we know the values of the user space registers EIP, ESP and EBP (for example by inspecting the `istate` structure as shown above). We can load the user space symbols:
    147147
    148148{{{
    149 $ objdump -h uspace/app/tester/tester
    150 
    151 uspace/app/tester/tester:     file format elf32-i386
    152 
    153 Sections:
    154 Idx Name          Size      VMA       LMA       File off  Algn
    155   0 .init         0000002e  000010b4  000010b4  000000b4  2**0
    156                   CONTENTS, ALLOC, LOAD, READONLY, CODE
    157   1 .text         0002f5a0  000010e8  000010e8  000000e8  2**3
    158                   CONTENTS, ALLOC, LOAD, READONLY, CODE
    159   2 .data         00001b08  000316a0  000316a0  0002f6a0  2**5
    160                   CONTENTS, ALLOC, LOAD, DATA
    161   3 .tbss         00000048  000331a8  000331a8  000311a8  2**2
    162                   ALLOC, THREAD_LOCAL
    163   4 .bss          00000184  000331c0  000331c0  000311a8  2**5
    164                   ALLOC
    165   5 .comment      00000011  00000000  00000000  000311a8  2**0
    166                   CONTENTS, READONLY
    167   6 .debug_abbrev 0000878f  00000000  00000000  000311b9  2**0
    168                   CONTENTS, READONLY, DEBUGGING
    169   7 .debug_aranges 00000db8  00000000  00000000  00039948  2**3
    170                   CONTENTS, READONLY, DEBUGGING
    171   8 .debug_info   000293a5  00000000  00000000  0003a700  2**0
    172                   CONTENTS, READONLY, DEBUGGING
    173   9 .debug_line   0000cfc7  00000000  00000000  00063aa5  2**0
    174                   CONTENTS, READONLY, DEBUGGING
    175  10 .debug_ranges 00000358  00000000  00000000  00070a6c  2**0
    176                   CONTENTS, READONLY, DEBUGGING
    177  11 .debug_str    000078fc  00000000  00000000  00070dc4  2**0
    178                   CONTENTS, READONLY, DEBUGGING
     149(gdb) add-symbol-file dist/app/tester
     150add symbol table from file "dist/app/tester"
     151(y or n) y
     152Reading symbols from dist/app/tester...done.
    179153}}}
    180154
    181 So the `.text` section gets loaded at 0x10e8 in this case. We will use this address to load our userspace symbols:
    182 
    183 {{{
    184 (gdb) add-symbol-file uspace/app/tester/tester 0x000010e8
    185 add symbol table from file "uspace/app/tester/tester" at
    186         .text_addr = 0x10e8
    187 (y or n) y
    188 Reading symbols from uspace/app/tester/tester...done.
    189 }}}
    190 
    191 The last step before we can print out our userspace stack trace is restoring registers to their userspace contents (as for example captured in the `istate` structure):
     155The last step before we can print out our userspace stack trace is restoring registers to their user space contents (as for example captured in the `istate` structure):
    192156
    193157{{{
     
    197161}}}
    198162
    199 We are now ready to do some userspace debugging:
     163We are now ready to do some user space debugging:
    200164
    201165{{{
     
    223187
    224188These two snippets above confirm that the page fault exception was indeed deliberately injected by the `tester` application when running test `fault1`.
     189
     190== Interactive user space debugging ==
     191
     192While using GDB outside QEMU to interactively debug an user space task in QEMU is tricky, since the GDB does not understand the notion of tasks and threads within the virtual machine, we can achieve it at least to a certain degree. First we need to make sure that we are running in the proper user space task context, for example by setting a conditional kernel breakpoint like this:
     193
     194{{{
     195(gdb) hbreak as_switch:1690 if $_streq(CURRENT->task->name, "/app/tester")
     196Hardware assisted breakpoint 1 at 0x801199b3: file ../../HelenOS/kernel/generic/src/mm/as.c, line 1623.
     197}}}
     198
     199Note that the breakpoint position should be the last statement of the `as_switch()` function. When the breakpoint hits, we are safe to assume that we running in the address space of `/app/tester` and we can setup additional breakpoints in user space. However, during the single-stepping of the user space task we need to be prepared for the preemption to kernel space (and eventually to a different task) at any time due to exception and interrupt handling.