Splunk is down

strive — Mon, 28 Sep 2020 14:03:29 GMT

Hi,

We have our application running on RHEL. All of sudden it stopped working and did not allow users to login, we were getting error Splunkd daemon is not responding: ('[Errno 111] Connection refused'). When we checked node by node, we noticed that on search head splunkd was not running. We restarted splunk on search head node and everything started functioning as usual.

When we checked logs (splunkd, splunkd_stderr, web_access, web_service, crash) we just found following error or warning messages at different instances. Other than these, nothing else was there.

06-03-2013 02:03:31.849 +0200 WARN  AuthenticationManagerScripted - Function 'getUsers' failed. Could not find '--status=success' in output
06-03-2013 02:03:31.849 +0200 ERROR AuthenticationManagerScripted - Script function getUsers failed

06-04-2013 07:16:47.423 +0200 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SessionManagerStatistics/bin/webservice1.py" INFO:root:Error: <urlopen error [Errno -3] Temporary failure in name resolution>

06-04-2013 09:48:23.507 +0200 ERROR HTTPClient - Cannot find host "splunkbase.splunk.com": Name or service not known
06-04-2013 09:48:23.507 +0200 ERROR ApplicationUpdater - Error checking for update via https://splunkbase.splunk.com/api/apps:resolve/checkforupgrade: Invalid URI

06-04-2013 16:44:10.005 +0200 WARN  EventLoop - Main Thread: about to throw a EventLoopException: error from PolledSocket write: Broken pipe

What could be the issue?

Thanks

Strive

Re: Splunk is down

MHibbin — Mon, 10 Jun 2013 08:43:40 GMT

Are you Splunking data from the search head OS? - might be worth checking the usual stats from there, as it could be relating to the OS instead of just Splunk.

Re: Splunk is down

ShaneNewman — Mon, 10 Jun 2013 13:14:35 GMT

We had a similar problem a few months ago. Turned out that it was not a Splunk problem, instead an AD issue. AD was not sending all of the data back that was requested.

There could be another issue though. It almost seems as if Splunk is not indexing the data coming back in a single event from what you say.

topic Splunk is down in Monitoring Splunk

Splunk is down

Re: Splunk is down

Re: Splunk is down