Deployment Architecture

What will cause AIX get core dump when Splunk Light Weight Forwarder restarts?

Splunk Employee
Splunk Employee

Sometimes when restart the Splunk Light Forwarder, user will experience a core dump. The forwarder still restarts and functions properly, but the core dump will fill up user's root filesystem.

The problem isn't limited to one host; it happens on several hosts, but all are running AIX 5.3-09.

Tags (1)
1 Solution

SplunkTrust
SplunkTrust

You might check aix's error logs (use the errpt command) - usually a core dump is logged there, with as much information as AIX can figure out.

Often, figuring this kind of thing out would mean having to have:

  • a copy of the binary that hasn't been stripped
  • the source code
  • an AIX machine at the exact same level of libc and kernel
  • the 'full' core file (run "chdev -l sys0 -a fullcore=true") and reboot

In other words, you'll probably need to engage Splunk support.

You might do as well to just disable corefiles for splunk. The only way (for sure) that I know how to do it would be to move $SPLUNK_HOME/bin/splunk to $SPLUNK_HOME/bin/splunk_real and make a shell script to replace the splunk binary, something like:

#!/bin/ksh
ulimit -c 0  #disable core dumps
exec splunk_real $0

View solution in original post

SplunkTrust
SplunkTrust

You might check aix's error logs (use the errpt command) - usually a core dump is logged there, with as much information as AIX can figure out.

Often, figuring this kind of thing out would mean having to have:

  • a copy of the binary that hasn't been stripped
  • the source code
  • an AIX machine at the exact same level of libc and kernel
  • the 'full' core file (run "chdev -l sys0 -a fullcore=true") and reboot

In other words, you'll probably need to engage Splunk support.

You might do as well to just disable corefiles for splunk. The only way (for sure) that I know how to do it would be to move $SPLUNK_HOME/bin/splunk to $SPLUNK_HOME/bin/splunk_real and make a shell script to replace the splunk binary, something like:

#!/bin/ksh
ulimit -c 0  #disable core dumps
exec splunk_real $0

View solution in original post

SplunkTrust
SplunkTrust

no, it would not disable core drumps for all root processes. Again, I would strongly suggest getting Splunk support involved if you are getting core dumps regularly. My suggestion above was intended to be a temporary workaround to supress corefiles until Splunk support could figure out the root cause and fix it.

0 Karma

Splunk Employee
Splunk Employee

will this script disable core dumps for all root processes?

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!