Using KGDB with VMWare

KGDB is a set of kernel patches that allows a second machine running a debugger (gdb) to control the target machine over a serial port. This page describes:

• compiling and running KGDB-enabled kernels, for versions 2.4 and 2.6.
• establishing a serial connection between gdb and a virtual machine running under Windows or Linux

---

Compiling and running a 2.4 kernel with KGDB

1. Download the kgdb 1.9 kernel patch for 2.4.23 from: linux-2.4.23-kgdb-1.9.patch
2. Apply the patch:

     cd /usr/src/linux-2.4.23   patch -p1 < /tmp/linux-2.4.23-kgdb-1.9.patch 

3. make menuconfig, and under "Kernel Hacking" enable "KGDB" and "KGDB Thread Analysis"
4. build your kernel. (since it's 2.4, that means make dep and make bzImage)
5. copy the kernel (bzImage) to the target virtual machine, or alternately copy vmlinux, the kernel executable, to the debugger machine.
6. add another lilo entry for the debug kernel, and append the argument "gdb" to get it to stop in the debugger during boot. I added the following lines to my lilo.conf file:

   prompt timeout=100 image=/vmlinuz.kgdb         label=KGDB         root=/dev/hda2         read-only         append="gdb gdbbaud=115000" 

   
   
   Note that the default image is the first in the file; the "prompt" and "timeout" arguments will allow you to specify which image to use at boot time. Don't forget to run lilo so it can save the parameters from /etc/lilo.conf.
7. Boot. It should output about 20 lines and then stop, waiting for the debugger.

Compiling and running a 2.6 kernel with KGDB

1. get the kgdb patches for kernel 2.6.15 here: http://kgdb.linsyssoft.com/downloads/kgdb-2/linux-2.6.15.5-kgdb-2.4.tar.bz2
2. untar them and then apply them:

     cd linux-2.6.15.6   patch -b -p1 < ../linux-2.6.15.5-kgdb-2.4/core-lite.patch   patch -b -p1 < ../ ... core.patch   patch -b -p1 < ../ ... i386-lite.patch                          i386.patch                          8250.patch 

3. configure and build your kernel. Under "kernel hacking" enable kgdb, set serial port to "0" (i.e. /dev/ttyS0, or COM1 - the default is ttyS1), and enable kernel debug info.
4. Add the command "kgdbwait" to the kernel command line for the kgdb kernel, and boot it. After printing "Booting the kernel" it should hang.

Connecting to VMWare KGDB, Linux host

1. Power down your virtual machine and click "Edit Virtual Machine Settings". Add a serial port, select "Connect to named pipe", and provide the pipe name "/tmp/com_1". (note that this isn't really a named pipe, but a unix-domain socket) Click "advanced settings" and set "yield CPU on poll". You can also get to this setting from the device list once you finish adding the port.
2. GDB can't connect directly to a unix-domain socket, so you'll need to use the socat ("socket cat") utility. It's installed on the elab machines, or you can install it from FC5 RPM or CentOS 4 RPM.
3. If the socket created by VMWare is /tmp/com_1, then invoke socat as: socat -d -d /tmp/com_1 PTY:
4. Find the line in the output identifying the PTY that was created:

   -bash-3.00$ socat -d -d /tmp/com_2 PTY: 2006/10/17 10:44:46 socat[20383] N successfully connected via  2006/10/17 10:44:46 socat[20383] N PTY is '''/dev/pts/7''' 2006/10/17 10:44:46 socat[20383] N starting data transfer loop ... 

5. Start gdb on the vmlinux executable, and connect to the target:

   -bash-3.00$ gdb vmlinux.kgdb  GNU gdb Red Hat Linux (6.3.0.0-1.96rh)   [...] (gdb) target remote /dev/pts/7 Remote debugging using /dev/pts/7 0xc012c138 in .text.lock.kgdbstub () at kgdbstub.c:1007 1007    kgdbstub.c: No such file or directory.         in kgdbstub.c warning: shared library handler failed to enable breakpoint (gdb) 

And now you're up and running.

Connecting to KGDB, Windows Hhost

1. Add 2 virtual serial ports to your virtual machine. Configure the second one to connect to a named pipe named \.pipecom_2. Under advanced settings, check the box for "yield CPU on poll"
2. Download and install the Named Pipe TCP Proxy utility: nptp_setup.zip
3. Start the proxy, and establish a connector between some port on the local machine (e.g. 8111) and \.pipecom_2. Check the box to enable non-local systems access.
4. In another virtual machine run 'gdb vmlinux'. (you'll have to work out how to share the source tree between the two VMs - clearly you can't use NFS from the VM that you're debugging)
5. type target remote ip_addr:8111 where ip_addr is the windows host IP address that this virtual machine sees. (e.g. on my machine 192.168.149.1, and the VM gets address 192.168.149.128)

KGDB should connect, and away you go.

Caveats - note that I've had a lot of trouble with the DHCP server in VMWare if I'm running multiple machines at once, as it tends to give the same address to each one. You may wish to hardcode the ip address, route, and DNS resolver instead on each VM.

Debugging modules under KGDB

1. get the address of the running module by running 'cat /proc/modules' on the target machine.
2. in gdb, type add-symbol-file /path/to/module/object.o 0xADDRESS which will load the symbols.