Forwarding Search Heads Crash Often Since 4.3 -- Anyone else?

[twinspop]

Anyone else getting dead splunkds? Unfortunately, the splunkd log isn't giving any useful info. Tips on where else to look? I've got 2 totally separate SHs that are doing this. Both do forwarding duties as well as SH duties. Prior to 4.3 I don't recall ever seeing them crash.

Linux x64

[dart]

I've seen something that looks similar to this. Are you using LDAP authentication and do you have referrals enabled? It seems to be working properly with this disabled (in this case, referrals are not required).

[twinspop]

No LDAP here, nor referrals. The crashes have seemed to stop with 4.3.1 and later.

[mwagstaff]

Just to chip in here... I've also seen this on one of my search heads (also Linux x64) since upgrading one of our environments the other day. I couldn't find any obvious causes in splunkd.log, and had a hunt around in Splunk on Splunk to no avail.

I haven't raised a support case yet as it's not a production environment and was only a once-off but if it re-occurs, I'll do so as it would be good to understand what's going on (hopefully without upping all log levels from info to debug).

[dwaddle]

Is Splunkd dying entirely, or just search processes? It's possible you could be hitting http://splunk-base.splunk.com/answers/37809/halp-consulting-the-summary-dashboard-of-the-search-app-... . But, that issue won't cause the "main" splunkd to die, just search processes.

[dwaddle]

Then, nope, wrong answer!

[twinspop]

When trying to contact the search head: "The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running." And when I issue a restart: "splunkd somepid was not running." Thanks for heads-up tho. I had noticed that other problem as well!