Debugging a Segfault

There's a brief page on cfengine segfaults at http://www.cfengine.org/confdir/segfault.html but it doesn't describe 
"post facto" debugging. As it is not usually feasible to run production services under the debugger, usually all we're
left with is a corefile. Let me emphasise this first:

'''DO NOT EMAIL COREFILES TO THE MAILING LIST!'''

A corefile is not helpful to anyone except you, because nobody else has access to the binary that generated the corefile.


== How to get a useful core file ==

First, if you're using a package manager (such as RPM or pkgadd), make sure the build process a) preserves the '-g' flag while compiling; and b) does not use strip(1) on the compiled binaries. For the most recent Red Hat-based distributions, the debugging info is stripped from the main binaries
but saved in a separate package named <code>packagename-debuginfo-<i>version.arch</i>.rpm</code>

Next, make sure your shell is configured to allow core files to be created. A common thing to do for security purposes is to remove the ability to create core files.  In our case, we ''want'' core files to be produced.  Note that this can be done easily and without altering the system values that were previously set.

To do this in almost all shells, the command <code>ulimit</code> controls whether core files will be generated.  Use the command
 ulimit -c unlimited
(which sets the core file maximum size), then use the command
 ulimit -a
to verify that the core files are indeed set to be generated. Start up the <code>cfservd</code> or <code>cfagent</code> from this shell directly -
if you use a startup script, the <code>ulimit</code> value may not be inherited into its environment.

Note that the <code>ulimit</code> command is quite often a shell built-in, but is sometimes a system binary (if there is no shell built-in).  Note, too, that this setting is inherited by spawned shells; thus, if a shell changes the values set by <code>ulimit</code>, then all future shells will have that new value.

Your core file will be generated in the <code>cwd</code> (or ''current working directory'') of the binary - you can use a tool like lsof(8) to find out what that is, or look at the <code>/proc</code> entry for the pid of your binary, if your operating system has a <code>procfs</code> filesystem.

To test that a core file will be generated the way you expect, you can <code>kill -SEGV</code> your process.  You should end up with a nice core file, suitable for running <code>gdb</code> on.

== What to do once you have a corefile ==

Invoke gdb on your corefile with the '-c' flag

 [ Copyright snipped ]
 gdb -c /path/to/corefile
 Program terminated with signal 11, Segmentation fault.
 #0  0xb73853b8 in ?? ()
 (gdb) 

At the (gdb) prompt, load the binary which generated the core with the 'file' command:

 (gdb) file /usr/sbin/cfservd
 Reading symbols from /usr/sbin/cfservd...Reading symbols from /usr/lib/debug//usr/sbin/cfservd.debug...done.
 Using host libthread_db library "/lib/tls/libthread_db.so.1".
 done.

Now use the 'bt' (backtrace) command to display what the program was doing when it crashed:
 (gdb) bt
 #0  0x08064976 in DeleteItemGeneral (list=0x80924b0, string=0x405f2af0 "10.10.16.209", type=regexComplete)
    at item-ext.c:707
 #1  0x4707521c in ?? ()
 #2  0x08064baf in DeleteItemMatching (list=0x80924b0, string=0x405f2af0 "10.10.16.209") at item-ext.c:767
 #3  0x08050a28 in DeleteConn (conn=0x1) at cfservd.c:3268
 #4  0x0804c6fb in HandleConnection (conn=0x40508b50) at cfservd.c:1129
 #5  0x401d32b6 in ?? ()

The above should be enough to pinpoint the problem, but optionally the 'bt full' command can give more information.

Mail that output to the list and, as Mark says -- Are you running the latest version?