Getting Data In

Why is our Splunk server crashing with error "Received fatal signal 11 (Segmentation fault)...No memory mapped"?

laquerre007
New Member

OS : Centos 6.7
Splunk Version : 6.3.2

For a few months our Splunk server keeps on crashing every 15 minutes or so
When verifying the splunkd logs, here are the details of what I saw:

Received fatal signal 11 (Segmentation fault).
 Cause:
   No memory mapped at address [0x00000054].
 Crashing thread: IndexerTPoolWorker-1

Any clue as to why this is happening?

0 Karma

Janssen135
Loves-to-Learn

Any solution for the above issue? I have the same one in Splunk version 8.1.6

0 Karma

mrgibbon
Contributor

I managed to work around this by un-taring the current version of Splunk over the top of the installation.
Running a chown command to make sure the files were all owned by the right user, then starting up again.
Worked for me, hope this can help someone else.

0 Karma

selim
Path Finder

Hi,
We have been facing the exact same issue. Interestingly enough, we were able to replicate the issue by simply opening up a dashboard and separated the search head and indexer to figure out where the problem was. Search Head was crashing with the existing configuration.

Short story:
We found a savedsearch within a user's context (private) that was named as a single character "a". Once this saved search was renamed to something longer, the problem went away.

$SPLUNK_HOME/etc/users/mary/search/local/savedsearches.conf
[a]
...

rename the search name to be something longer:

[some_longer_name_a]
...

For this we had to edit the file, you can not do this from the web interface.

Long story:
The problem occurred when one of the available dashboards opened (or tried to open the link). This also happened when we create a very simple dashboard with one simple search panel. We were not able to replicate it with concurrent searches so this very much seemed like an issue with web instance.

Splunk crashed within the same place all the time and the issue was replicated easily. Here's a portion of the crash log:

[build aaff59bb082c] 2016-01-29 21:18:31
Received fatal signal 11 (Segmentation fault).
Cause:
   No memory mapped at address [0x0000000000000008].
Crashing thread: TcpChannelThread
Registers:
    RIP:  [0x0000000000DA1D78] _ZNK9Paginator3cmpEP10ConfigItemS1_m + 104 (splunkd)
...
    OLDMASK:  [0x0000000000000000]
OS: Linux
Arch: x86-64

On a brand new search head, we added apps ($SPLUNK_HOME/etc/apps) and local config ($SPLUNK_HOME/etc/system/local) and user config ($SPLUNK_HOME/etc/users) one by one to figure out where the problem may be.

It boiled down to one specific user configuration, say "mary" ($SPLUNK_HOME/etc/users/mary). So we one by one removed existing configuration for that user: dashboards, panels, and configuration files and tested the search head crash (opening up a dashboard). It turned out to be the savedsearches.conf file as mentioned in the short version of this story above.

The other interesting finding is that when we logon as "mary" and open up this private dashboard, nothing bad happens, no crashes.

Conclusion:
There's a ticket opened up and we still do not have a fix for this issue yet, but we were able to find out that some users were not following the naming conventions 🙂

Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...