Why is our Splunk server crashing with error "Received fatal signal 11 (Segmentation fault)...No memory mapped"?

[laquerre007]

OS : Centos 6.7
Splunk Version : 6.3.2

For a few months our Splunk server keeps on crashing every 15 minutes or so
When verifying the splunkd logs, here are the details of what I saw:

Received fatal signal 11 (Segmentation fault).
 Cause:
   No memory mapped at address [0x00000054].
 Crashing thread: IndexerTPoolWorker-1

Any clue as to why this is happening?

[Janssen135]

Any solution for the above issue? I have the same one in Splunk version 8.1.6

[mrgibbon]

I managed to work around this by un-taring the current version of Splunk over the top of the installation.
Running a chown command to make sure the files were all owned by the right user, then starting up again.
Worked for me, hope this can help someone else.

[selim]

Hi,
We have been facing the exact same issue. Interestingly enough, we were able to replicate the issue by simply opening up a dashboard and separated the search head and indexer to figure out where the problem was. Search Head was crashing with the existing configuration.

Short story:
We found a savedsearch within a user's context (private) that was named as a single character "a". Once this saved search was renamed to something longer, the problem went away.

$SPLUNK_HOME/etc/users/mary/search/local/savedsearches.conf
[a]
...

rename the search name to be something longer:

[some_longer_name_a]
...

For this we had to edit the file, you can not do this from the web interface.

Long story:
The problem occurred when one of the available dashboards opened (or tried to open the link). This also happened when we create a very simple dashboard with one simple search panel. We were not able to replicate it with concurrent searches so this very much seemed like an issue with web instance.

Splunk crashed within the same place all the time and the issue was replicated easily. Here's a portion of the crash log:

[build aaff59bb082c] 2016-01-29 21:18:31
Received fatal signal 11 (Segmentation fault).
Cause:
   No memory mapped at address [0x0000000000000008].
Crashing thread: TcpChannelThread
Registers:
    RIP:  [0x0000000000DA1D78] _ZNK9Paginator3cmpEP10ConfigItemS1_m + 104 (splunkd)
...
    OLDMASK:  [0x0000000000000000]
OS: Linux
Arch: x86-64

On a brand new search head, we added apps ($SPLUNK_HOME/etc/apps) and local config ($SPLUNK_HOME/etc/system/local) and user config ($SPLUNK_HOME/etc/users) one by one to figure out where the problem may be.

It boiled down to one specific user configuration, say "mary" ($SPLUNK_HOME/etc/users/mary). So we one by one removed existing configuration for that user: dashboards, panels, and configuration files and tested the search head crash (opening up a dashboard). It turned out to be the savedsearches.conf file as mentioned in the short version of this story above.

The other interesting finding is that when we logon as "mary" and open up this private dashboard, nothing bad happens, no crashes.

Conclusion:
There's a ticket opened up and we still do not have a fix for this issue yet, but we were able to find out that some users were not following the naming conventions 🙂