Deployment Architecture

How to troubleshoot why the Splunk 7.1.x search head crashes periodically?

Splunk Employee
Splunk Employee

Splunk search head crashes periodically, Splunk version is 7.1. Below is the crash logs:

Received fatal signal 6 (Aborted).
Cause:
 Signal sent by PID 14764 running under UID 2000.
Crashing thread: SHPHeartbeatThread
Registers:
RIP: xxxxxxxxxxxxxxxxxxxxxxxx
  RDI:  xxxxxxxxxxxxxxxxxxxxxxxxx
  RSI: xxxxxxxxxxxxxxxxxxxxxxxx
  RBP:  xxxxxxxxxxxxxxxxxxxxxxx
  RSP:  xxxxxxxxxxxxxxxxxxxxxx
  RAX:  xxxxxxxxxxxxxxxxxxxxxxxx

ERROR - Failed opening "": No such file or directory
  Conf mutator lockfile has disappeared; error condition possible.
  Bypassing local license checks since this instance is configured with a remote license master.
  xxxxxxxxx 12:07:25.476 +0800 splunkd started (build 8f0ead9ec3db)
  splunkd: /home/build/build-src/nightlight/src/framework/SearchResultsMem.cpp:839: SearchResultsMem::iterator SearchResultsMem::erase(SearchResultsMem::iterator, SearchResultsMem::iterator): Assertion `it != end()' failed.
xxxxxxxxxxxx 10:46:31.839 +0800 splunkd started (build 8f0ead9ec3db)
  splunkd: /home/build/build-src/nightlight/src/framework/SearchResultsMem.cpp:839: SearchResultsMem::iterator SearchResultsMem::erase(SearchResultsMem::iterator, SearchResultsMem::iterator): Assertion `it != end()' failed.
0 Karma
1 Solution

Splunk Employee
Splunk Employee

Hi,

This is a know bug in Splunk version 7.1. If below is the reason for crash then it is a code bug and already fixed in Splunk 7.2.

splunkd: /home/build/build-src/nightlight/src/framework/SearchResultsMem.cpp:839: SearchResultsMem::iterator SearchResultsMem::erase(SearchResultsMem::iterator, SearchResultsMem::iterator): Assertion `it != end()' failed.

View solution in original post

Splunk Employee
Splunk Employee

Hi,

This is a know bug in Splunk version 7.1. If below is the reason for crash then it is a code bug and already fixed in Splunk 7.2.

splunkd: /home/build/build-src/nightlight/src/framework/SearchResultsMem.cpp:839: SearchResultsMem::iterator SearchResultsMem::erase(SearchResultsMem::iterator, SearchResultsMem::iterator): Assertion `it != end()' failed.

View solution in original post

Explorer

Do you know what is the Issue number of this bug?

0 Karma

SplunkTrust
SplunkTrust
  1. Look for that assertion in any known issues in the splunk docs
  2. Upgrade to the latest patch level in your version (if you are in Splunk 7.1 release train, then 7.1.x where x is the most recent)
  3. Submit a support case

When splunk exits on an assertion failure, this is it saying that a internal constraint in the code has been violated. Usually, these require access to the source code to understand and require a code fix to resolve. If it's not been resolved already, then you will need a support case.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!