Splunk is down

[strive]

Hi,

We have our application running on RHEL. All of sudden it stopped working and did not allow users to login, we were getting error Splunkd daemon is not responding: ('[Errno 111] Connection refused'). When we checked node by node, we noticed that on search head splunkd was not running. We restarted splunk on search head node and everything started functioning as usual.

When we checked logs (splunkd, splunkd_stderr, web_access, web_service, crash) we just found following error or warning messages at different instances. Other than these, nothing else was there.

06-03-2013 02:03:31.849 +0200 WARN  AuthenticationManagerScripted - Function 'getUsers' failed. Could not find '--status=success' in output
06-03-2013 02:03:31.849 +0200 ERROR AuthenticationManagerScripted - Script function getUsers failed

06-04-2013 07:16:47.423 +0200 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SessionManagerStatistics/bin/webservice1.py" INFO:root:Error: <urlopen error [Errno -3] Temporary failure in name resolution>

06-04-2013 09:48:23.507 +0200 ERROR HTTPClient - Cannot find host "splunkbase.splunk.com": Name or service not known
06-04-2013 09:48:23.507 +0200 ERROR ApplicationUpdater - Error checking for update via https://splunkbase.splunk.com/api/apps:resolve/checkforupgrade: Invalid URI

06-04-2013 16:44:10.005 +0200 WARN  EventLoop - Main Thread: about to throw a EventLoopException: error from PolledSocket write: Broken pipe

What could be the issue?

Thanks

Strive

[ShaneNewman]

We had a similar problem a few months ago. Turned out that it was not a Splunk problem, instead an AD issue. AD was not sending all of the data back that was requested.

There could be another issue though. It almost seems as if Splunk is not indexing the data coming back in a single event from what you say.

[MHibbin]

Are you Splunking data from the search head OS? - might be worth checking the usual stats from there, as it could be relating to the OS instead of just Splunk.