Changes between Version 4 and Version 5 of DebuggingWithQEMUAndGDB
- Timestamp:
- 2020-01-14T15:36:41Z (4 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
DebuggingWithQEMUAndGDB
v4 v5 6 6 }}} 7 7 8 The combination of QEMU and GDB allows HelenOS to be comfortably debugged either on the assembly or the source code level. For detailed information on low-level debugging, see for example this [http://d3s.mff.cuni.cz/teaching/crash_dump_analysis/ course] on crash dump analysis.8 The combination of QEMU and GDB allows HelenOS to be comfortably debugged either on the assembly or the source code level. 9 9 10 10 == Preparing the build == 11 11 12 The default HelenOS build should produce unstripped binaries. If necessary, this can be enforced by making sure the `Strip binaries` build configuration option is not checked. Unstripped binaries come with symbols, but do not contain any fancier debugging information. In order to get maximum out of the debug build, make sure to configure HelenOS with the `Line debugging information` option. Another thing which may impede debugging is optimization, so consider changing optimization levels to 0 using the `O PTIMIZATION` variable in the respective Makefiles ([browser:mainline/kernel/Makefile] or [browser:mainline/uspace/Makefile.common]). When everything is set, make sure to rebuild with the new settings.12 The default HelenOS build should produce unstripped binaries. If necessary, this can be enforced by making sure the `Strip binaries` build configuration option is not checked. Unstripped binaries come with symbols, but do not contain any fancier debugging information. In order to get maximum out of the debug build, make sure to configure HelenOS with the `Line debugging information` option. Another thing which may impede debugging is optimization, so consider changing optimization levels to 0 using the `Optimization level` configuration option. When everything is set, make sure to rebuild with the new settings. 13 13 14 14 == Starting QEMU == 15 15 16 QEMU provides two command line options for debugging with GDB: `-s` and `-S`. The former instructs QEMU to listen for GDB connections on localhost:1234 (but does not wait for it) and the latter stops the guest CPU at startup so that debugging is possible from the very beginning. When starting emulation using the [browser:mainline/tools/ew.py]script, one can add these options like this:16 QEMU provides two command line options for debugging with GDB: `-s` and `-S`. The former instructs QEMU to listen for GDB connections on localhost:1234 (but does not wait for it) and the latter stops the guest CPU at startup so that debugging is possible from the very beginning. When starting emulation using the `tools/ew.py` script, one can add these options like this: 17 17 18 18 {{{ … … 22 22 == Connecting GDB to QEMU == 23 23 24 Once QEMU is started with the `-s` (and optionally also the `-S`) option, it is possible to connect GDB to it. For our purposes, we will assume the respective cross-GDB built by the [browser:mainline/tools/toolchain.sh] is ready to be used and named as `/usr/local/cross/ia32/bin/i686-pc-linux-gnu-gdb`. Needless to say, the cross-GDB should always match the architecture of the HelenOS guest.24 Once QEMU is started with the `-s` (and optionally also the `-S`) option, it is possible to connect GDB to it. For our purposes, we will assume the respective cross-GDB built by the `tools/toolchain.sh` is ready to be used and named as `/usr/local/cross/bin/i686-helenos-gdb`. Needless to say, the cross-GDB should always match the architecture of the HelenOS guest. 25 25 26 26 {{{ 27 $ /usr/local/cross/ ia32/bin/i686-pc-linux-gnu-gdb27 $ /usr/local/cross/bin/i686-helenos-gdb 28 28 GNU gdb (GDB) 7.11 29 29 Copyright (C) 2016 Free Software Foundation, Inc. … … 51 51 }}} 52 52 53 Note that if QEMU was not started with the `-S` option, you will have to manually break into the debugger by pressing `Ctrl -C`.53 Note that if QEMU was not started with the `-S` option, you will have to manually break into the debugger by pressing `Ctrl+C`. 54 54 55 55 == Loading symbols == 56 56 57 Depending on what is the subject of our debugging session, we will need some symbols. The easiest case is kernel debugging. In that case, we simply load the kernel symbols from `kernel. raw`:57 Depending on what is the subject of our debugging session, we will need some symbols. The easiest case is kernel debugging. In that case, we simply load the kernel symbols from `kernel.elf`: 58 58 59 59 {{{ 60 (gdb) symbol-file kernel/kernel.raw61 Reading symbols from kernel/kernel.raw...done.60 (gdb) symbol-file dist/boot/kernel.elf 61 Reading symbols from dist/boot/kernel.elf...done. 62 62 }}} 63 63 64 Debugging user space tasks is a little more complicated because unlike the kernel, there will be multiple user processes running at the same time, so setting a mere breakpoint on a user address will probably not do. We will need to get some assistance from the kernel.64 Debugging user space tasks is a little more complicated because unlike the kernel, there will be multiple user processes running at the same time, so setting a mere breakpoint on a user address will probably not do. We will need to get some assistance from the kernel. 65 65 66 66 == Setting and hitting breakpoints == 67 67 68 To set a breakpoint, for example on the kernel function that handles invalid memory accesses from user space, we type the following command:68 To set a breakpoint, for example on the kernel function that handles invalid memory accesses from user space, we type the following command: 69 69 70 70 {{{ … … 74 74 75 75 {{{#!box type=warning 76 Sadly the set breakpoint is not always hit. At this point, the cause of this behavior is unknown.76 Note that the default breakpoint mechanism modifies the code running in QEMU by inserting a breakpoint instruction (`INT3` on ia32). Thus if you setup a breakpoint using the `break` command before the kernel boots or an user space task loads, the breakpoint instruction might get overwritten and the breakpoint rendered ineffective. To avoid this situation, you can use hardware-assisted breakpoints using the `hbreak` command. Unfortunately, the number of hardware-assisted breakpoints is limited. 77 77 }}} 78 78 … … 98 98 }}} 99 99 100 Note that at any time while the guest is running, you can break into the debugger also by pressing the `Ctrl -C` combo.100 Note that at any time while the guest is running, you can break into the debugger also by pressing the `Ctrl+C` combo. 101 101 102 102 Now that a breakpoint in the kernel was hit, we can inspect the state of the kernel a little bit. Typing `bt` will show us the stack trace of the current kernel thread: … … 127 127 == Useful macros == 128 128 129 When debugging the kernel, it is sometimes useful to find out some information about the current process. For that, we will need to mimic the computation of the address of the `THE` structure. We will start by defining a macro:129 When debugging the kernel, it is sometimes useful to find out some information about the current task. For that, we will need to mimic the computation of the address of the `CURRENT` structure. We will start by defining a macro: 130 130 131 131 {{{ 132 (gdb) macro define the ((the_t *) ((uintptr_t )$esp & ~0x1fff))132 (gdb) macro define CURRENT ((current_t *) ((uintptr_t) $esp & ~0x1fff)) 133 133 }}} 134 134 … … 136 136 137 137 {{{ 138 (gdb) p the->task->name139 $3 = "tester\000it\000\000\000\000\000\000\000\000\000\000"138 (gdb) x/s CURRENT->task->name 139 0x879748c8: "/app/tester" 140 140 }}} 141 141 … … 144 144 == Switching to the user context == 145 145 146 Let us assume that we have somehow found the values of the userspace registers EIP, ESP and EBP (for example by inspecting the `istate` structure as shown above) and know the name of the process (for example `tester` from the example above). Before we can add symbol information for this process, we need to find out the load address of its `.text` section (for some reason, GDB does not use the information provided in the ELF file):146 Let us assume that we know the values of the user space registers EIP, ESP and EBP (for example by inspecting the `istate` structure as shown above). We can load the user space symbols: 147 147 148 148 {{{ 149 $ objdump -h uspace/app/tester/tester 150 151 uspace/app/tester/tester: file format elf32-i386 152 153 Sections: 154 Idx Name Size VMA LMA File off Algn 155 0 .init 0000002e 000010b4 000010b4 000000b4 2**0 156 CONTENTS, ALLOC, LOAD, READONLY, CODE 157 1 .text 0002f5a0 000010e8 000010e8 000000e8 2**3 158 CONTENTS, ALLOC, LOAD, READONLY, CODE 159 2 .data 00001b08 000316a0 000316a0 0002f6a0 2**5 160 CONTENTS, ALLOC, LOAD, DATA 161 3 .tbss 00000048 000331a8 000331a8 000311a8 2**2 162 ALLOC, THREAD_LOCAL 163 4 .bss 00000184 000331c0 000331c0 000311a8 2**5 164 ALLOC 165 5 .comment 00000011 00000000 00000000 000311a8 2**0 166 CONTENTS, READONLY 167 6 .debug_abbrev 0000878f 00000000 00000000 000311b9 2**0 168 CONTENTS, READONLY, DEBUGGING 169 7 .debug_aranges 00000db8 00000000 00000000 00039948 2**3 170 CONTENTS, READONLY, DEBUGGING 171 8 .debug_info 000293a5 00000000 00000000 0003a700 2**0 172 CONTENTS, READONLY, DEBUGGING 173 9 .debug_line 0000cfc7 00000000 00000000 00063aa5 2**0 174 CONTENTS, READONLY, DEBUGGING 175 10 .debug_ranges 00000358 00000000 00000000 00070a6c 2**0 176 CONTENTS, READONLY, DEBUGGING 177 11 .debug_str 000078fc 00000000 00000000 00070dc4 2**0 178 CONTENTS, READONLY, DEBUGGING 149 (gdb) add-symbol-file dist/app/tester 150 add symbol table from file "dist/app/tester" 151 (y or n) y 152 Reading symbols from dist/app/tester...done. 179 153 }}} 180 154 181 So the `.text` section gets loaded at 0x10e8 in this case. We will use this address to load our userspace symbols: 182 183 {{{ 184 (gdb) add-symbol-file uspace/app/tester/tester 0x000010e8 185 add symbol table from file "uspace/app/tester/tester" at 186 .text_addr = 0x10e8 187 (y or n) y 188 Reading symbols from uspace/app/tester/tester...done. 189 }}} 190 191 The last step before we can print out our userspace stack trace is restoring registers to their userspace contents (as for example captured in the `istate` structure): 155 The last step before we can print out our userspace stack trace is restoring registers to their user space contents (as for example captured in the `istate` structure): 192 156 193 157 {{{ … … 197 161 }}} 198 162 199 We are now ready to do some user space debugging:163 We are now ready to do some user space debugging: 200 164 201 165 {{{ … … 223 187 224 188 These two snippets above confirm that the page fault exception was indeed deliberately injected by the `tester` application when running test `fault1`. 189 190 == Interactive user space debugging == 191 192 While using GDB outside QEMU to interactively debug an user space task in QEMU is tricky, since the GDB does not understand the notion of tasks and threads within the virtual machine, we can achieve it at least to a certain degree. First we need to make sure that we are running in the proper user space task context, for example by setting a conditional kernel breakpoint like this: 193 194 {{{ 195 (gdb) hbreak as_switch:1690 if $_streq(CURRENT->task->name, "/app/tester") 196 Hardware assisted breakpoint 1 at 0x801199b3: file ../../HelenOS/kernel/generic/src/mm/as.c, line 1623. 197 }}} 198 199 Note that the breakpoint position should be the last statement of the `as_switch()` function. When the breakpoint hits, we are safe to assume that we running in the address space of `/app/tester` and we can setup additional breakpoints in user space. However, during the single-stepping of the user space task we need to be prepared for the preemption to kernel space (and eventually to a different task) at any time due to exception and interrupt handling.
