Deployment Architecture

Why am I receiving syslog errors that splunkd was killed by signal 8 (SIGFPE) on my two clustered indexers?

cmccormick
Explorer

I am receiving errors in my syslog showing that splunkd is crashing about every couple of minutes on my two clustered indexers. I have been researching the issue and have yet to determine the root cause. I have checked the Splunk logs and their are no errors being reported there either.

Here is the syslog entry:

Nov 23 00:04:39 den1-spkix-301 kernel: splunkd[30897] trap divide error ip:c36e19 sp:7f0eafbfba20 error:0 in splunkd[400000+1ade000]

The reason file in the abrt folder shows Process splunk/bin/splunkd was killed by signal 8 (SIGFPE)

This started shortly after a new input started being forwarded to the indexers.

Here are the details for my servers:

OS: CentOS 6.5
Architecture: x86_64
Kernel: 2.6.32-431.el6.x86_64
Splunk Version: Splunk 6.3.0 (build aa7d4b1ccb80)

Any help would be appreciated.

1 Solution

dwaddle
SplunkTrust
SplunkTrust

I would suggest a support case. SIGFPE is a floating point exception, and you can see in your logs there a "trap divide error". It sounds like this new input may have issues. Disable it, see what difference that makes on your crashing. But, regardless, submit a support case with a diag.

Also you could try upgrading to 6.3.1 - I don't see this as a known issue, but it never hurts to show support you've done all you can do before contacting them.

View solution in original post

cmccormick
Explorer

I found out that I had a search head on Splunk 6.2.1 that was added to the cluster. When I removed it, the errors stopped.

dwaddle
SplunkTrust
SplunkTrust

I would suggest a support case. SIGFPE is a floating point exception, and you can see in your logs there a "trap divide error". It sounds like this new input may have issues. Disable it, see what difference that makes on your crashing. But, regardless, submit a support case with a diag.

Also you could try upgrading to 6.3.1 - I don't see this as a known issue, but it never hurts to show support you've done all you can do before contacting them.

wplank
Path Finder

I would recommend to upgrade to the latest build (6.3.1), I would also recomment to update your system to the latests patch level.

Kind regards

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...