What will cause AIX get core dump when Splunk Light Weight Forwarder restarts?

[zliu]

Sometimes when restart the Splunk Light Forwarder, user will experience a core dump. The forwarder still restarts and functions properly, but the core dump will fill up user's root filesystem.

The problem isn't limited to one host; it happens on several hosts, but all are running AIX 5.3-09.

[dwaddle · Accepted solution]

You might check aix's error logs (use the errpt command) - usually a core dump is logged there, with as much information as AIX can figure out.

Often, figuring this kind of thing out would mean having to have:

• a copy of the binary that hasn't been stripped
• the source code
• an AIX machine at the exact same level of libc and kernel
• the 'full' core file (run "chdev -l sys0 -a fullcore=true") and reboot

In other words, you'll probably need to engage Splunk support.

You might do as well to just disable corefiles for splunk. The only way (for sure) that I know how to do it would be to move $SPLUNK_HOME/bin/splunk to $SPLUNK_HOME/bin/splunk_real and make a shell script to replace the splunk binary, something like:

#!/bin/ksh
ulimit -c 0  #disable core dumps
exec splunk_real $0

[dwaddle]

no, it would not disable core drumps for all root processes. Again, I would strongly suggest getting Splunk support involved if you are getting core dumps regularly. My suggestion above was intended to be a temporary workaround to supress corefiles until Splunk support could figure out the root cause and fix it.

[zliu]

will this script disable core dumps for all root processes?