topic Re: What will cause AIX get core dump when Splunk Light Weight Forwarder restarts? in Deployment Architecture

What will cause AIX get core dump when Splunk Light Weight Forwarder restarts?

zliu — Tue, 25 May 2010 05:34:04 GMT

Sometimes when restart the Splunk Light Forwarder, user will experience a core dump. The forwarder still restarts and functions properly, but the core dump will fill up user's root filesystem.

The problem isn't limited to one host; it happens on several hosts, but all are running AIX 5.3-09.

Re: What will cause AIX get core dump when Splunk Light Weight Forwarder restarts?

dwaddle — Wed, 26 May 2010 05:01:49 GMT

You might check aix's error logs (use the errpt command) - usually a core dump is logged there, with as much information as AIX can figure out.

Often, figuring this kind of thing out would mean having to have:

a copy of the binary that hasn't been stripped
the source code
an AIX machine at the exact same level of libc and kernel
the 'full' core file (run "chdev -l sys0 -a fullcore=true") and reboot

In other words, you'll probably need to engage Splunk support.

You might do as well to just disable corefiles for splunk. The only way (for sure) that I know how to do it would be to move $SPLUNK_HOME/bin/splunk to $SPLUNK_HOME/bin/splunk_real and make a shell script to replace the splunk binary, something like:

#!/bin/ksh
ulimit -c 0  #disable core dumps
exec splunk_real $0

Re: What will cause AIX get core dump when Splunk Light Weight Forwarder restarts?

zliu — Wed, 02 Jun 2010 00:26:30 GMT

will this script disable core dumps for all root processes?

Re: What will cause AIX get core dump when Splunk Light Weight Forwarder restarts?

dwaddle — Wed, 02 Jun 2010 04:48:30 GMT

no, it would not disable core drumps for all root processes. Again, I would strongly suggest getting Splunk support involved if you are getting core dumps regularly. My suggestion above was intended to be a temporary workaround to supress corefiles until Splunk support could figure out the root cause and fix it.