|CRASH(8)||System Manager's Manual||CRASH(8)|
When the system crashes voluntarily it prints a message of the form
panic: why i gave up the ghost
on the console and enters the kernel debugger, ddb(4).
If you wish to report this panic, you should include the output of
Unless the ‘ddb.log’ sysctl has been disabled, anything output
to screen will be appended to the system message buffer, from where it may
be possible to retrieve it through the
dmesg(8) command after a warm
reboot. If the debugger command
boot dump is
entered, or if the debugger was not compiled into the kernel, or the
debugger was disabled with
sysctl(8), then the system
dumps the contents of physical memory onto a mass storage peripheral device.
The particular device used is determined by the ‘dumps on’
directive in the config(8)
file used to build the kernel.
After the dump has been written, the system then invokes the automatic reboot procedure as described in reboot(8). If auto-reboot is disabled (in a machine dependent way) the system will simply halt at this point.
Upon rebooting, and unless some unexpected inconsistency is encountered in the state of the file systems due to hardware or software failure, the system will copy the previously written dump into /var/crash using savecore(8), before resuming multi-user operations.
The most common cause of system failures is hardware failure (e.g., bad memory) which can reflect itself in different ways. Here are the messages which are most likely, with some hints as to causes. Left unstated in all cases is the possibility that a hardware or software error produced the message in some unexpected way.
The code is the referenced address, and the pc is the program counter at the time of the fault is printed. Hardware flakiness will sometimes generate this panic, but if the cause is a kernel bug, the kernel debugger ddb(4) can be used to locate the instruction and subroutine inside the kernel corresponding to the PC value. If that is insufficient to suggest the nature of the problem, more detailed examination of the system status at the time of the trap usually can produce an explanation.
That completes the list of panic types you are likely to see.
To analyze the kernel and memory images preserved as bsd.0 and bsd.0.core, you should run gdb(1), loading in the images with the following commands:
# gdb GNU gdb 6.3 Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-unknown-openbsd4.6". (gdb) file /var/crash/bsd.0 Reading symbols from /var/crash/bsd.0...(no debugging symbols found)...done. (gdb) target kvm /var/crash/bsd.0.core
[Note that the “kvm” target is currently only supported by gdb(1) on some architectures.]
After this, you can use the
to show trace of procedure calls that led to the crash.
For custom-built kernels, you should use bsd.gdb instead of bsd, thus allowing gdb(1) to show symbolic names for addresses and line numbers from the source.
Analyzing saved system images is sometimes called post-mortem debugging. There are a class of analysis tools designed to work on both live systems and saved images, most of them are linked with the kvm(3) library and share option flags to specify the kernel and memory image. These tools typically take the following flags:
The following commands understand these options:
vmstat(8) and many others.
There are exceptions, however. For instance,
ipcs(1) has renamed the
-M argument to be
Examples of use:
# ps -N /var/crash/bsd.0 -M /var/crash/bsd.0.core -O paddr
-O paddr option
prints each process'
struct proc address. This is
very useful information if you are analyzing process contexts in
# vmstat -N /var/crash/bsd.0 -M /var/crash/bsd.0.core -m
This analyzes memory allocations at the time of the crash. Perhaps some resource was starving the system?
(gdb) target kvm
It is possible to inspect processes that entered the kernel by
specifying a process'
struct proc address to the
kvm proc command:
(gdb) kvm proc 0xd69dada0 #0 0xd0355d91 in sleep_finish (sls=0x0, do_sleep=0) at ../../../../kern/kern_synch.c:217 217 mi_switch();
After this, the
where command will show a
trace of procedure calls, right back to where the selected process entered
First, in ddb(4) find
the function that caused the crash. It is either the function at the top of
the traceback or the function under the call to
The point of the crash usually looks something like this "function+0x4711".
Find the function in the sources, let's say that the function is in "foo.c".
Go to the kernel build directory, e.g., /sys/arch/ARCH/compile/GENERIC, and do the following:
# objdump -S foo.o | less
Find the function in the output. The function will look something like this:
0: 17 47 11 42 foo %x, bar, %y 4: foo bar allan %kaka 8: XXXX boink %bloyt etc.
The first number is the offset. Find the offset that you got in the ddb trace (in this case it's 4711).
When reporting data collected in this way, include ~20 lines before and ~10 lines after the offset from the objdump output in the crash report, as well as the output of ddb(4)'s "show registers" command. It's important that the output from objdump includes at least two or three lines of C code.
|November 29, 2016||OpenBSD-6.2|