Why am I receiving syslog errors that splunkd was killed by signal 8 (SIGFPE) on my two clustered indexers?

[cmccormick]

I am receiving errors in my syslog showing that splunkd is crashing about every couple of minutes on my two clustered indexers. I have been researching the issue and have yet to determine the root cause. I have checked the Splunk logs and their are no errors being reported there either.

Here is the syslog entry:

Nov 23 00:04:39 den1-spkix-301 kernel: splunkd[30897] trap divide error ip:c36e19 sp:7f0eafbfba20 error:0 in splunkd[400000+1ade000]

The reason file in the abrt folder shows Process splunk/bin/splunkd was killed by signal 8 (SIGFPE)

This started shortly after a new input started being forwarded to the indexers.

Here are the details for my servers:

OS: CentOS 6.5
Architecture: x86_64
Kernel: 2.6.32-431.el6.x86_64
Splunk Version: Splunk 6.3.0 (build aa7d4b1ccb80)

Any help would be appreciated.

[cmccormick]

I found out that I had a search head on Splunk 6.2.1 that was added to the cluster. When I removed it, the errors stopped.

[dwaddle · Accepted solution]

I would suggest a support case. SIGFPE is a floating point exception, and you can see in your logs there a "trap divide error". It sounds like this new input may have issues. Disable it, see what difference that makes on your crashing. But, regardless, submit a support case with a diag.

Also you could try upgrading to 6.3.1 - I don't see this as a known issue, but it never hurts to show support you've done all you can do before contacting them.

[wplank]

I would recommend to upgrade to the latest build (6.3.1), I would also recomment to update your system to the latests patch level.

Kind regards