Splunkd crash after upgrade to 4.3

[crossroadsIT]

After upgrading our splunk server to 4.3 from 4.2.5 splunkd crashes with the following errors in splunkd_stderr.log:

2012-01-11 11:01:27.098 +0800 splunkd started (build 115073)
terminate called after throwing an instance of 'PropertyPagesException'
what():  Cannot get user to act as: No user info provider registered (user: xxxx, app: user-prefs, root: /opt/splunk/etc)

Running on CentOS 5.7 64-bit, dual Quad Core Xeon with 8GB RAM.

[hexx · Accepted solution]

I believe you are experiencing a crash that was discovered just today and filed as a bug with reference SPL-47232.

Symptoms:

The signature of this crash is as follows :

• The crashing thread is always DispatchReaper, the thread which curates the search artifacts in the dispatch directory ($SPLUNK_HOME/var/run/splunk/dispatch). The crashing thread is indicated at the very beginning of the crash log file that can be found in $SPLUNK_HOME/var/log/splunk :
  

[build 115073] 2012-01-12 14:54:12
Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 28468 running under UID 0.
Crashing thread: DispatchReaperCrashing thread: DispatchReaper <==

• The file logging splunkd's stderr output ($SPLUNK_HOME/var/log/splunk/splunkd_stderr.log) will contain the following error:
  

terminate called after throwing an instance of 'PropertyPagesException'
what(): Cannot get user to act as: No user info provider registered (user: splunk-system-user, app: user-prefs, root: /opt/splunk/etc)

• The crash almost always occurs on splunkd start-up, sometimes on the first start-up attempt after the 4.3 upgrade. It has also been known to occur during normal operation of the Splunk instance.

Work-around:

Until this crash is fixed in an upcoming release, you'll have to take the following steps to allow splunkd to start again:

• Delete the contents of the search dispatch directory:

rm -rf $SPLUNK_HOME/var/run/splunk/dispatch/*

• Start up Splunk:

$SPLUNK_HOME/bin/splunk start

Note: If you are unfamiliar with the search dispatch directory, it is the location where Splunk stores search artifacts for past and currently running searches. That data is volatile by nature and can be regenerated by re-running the searches that generated it.

[crossroadsIT]

This is the exact error that we had faced. Interestingly our Splunk instance started working of its' own accord after a couple of hours.

[hexx]

We will at least need to look at the corresponding crash log and at what is logged at the time of the crash in splunkd.log in order to comment. I would strongly recommend that you log a support case and attach a Splunk diag to get this crash analyzed. It doesn't seem likely that we'll be able to determine the cause of the problem just from the information provided so far.