Splunkd daemon is not responding: ('The read operation timed out',)

[jangid]

Whenever I am searching anything I am getting
Splunkd daemon is not responding: ('The read operation timed out',)

I have restarted Splunk and I can see the data in main screen of Search App

any idea?

Thanks
Manoj Jangid

[sylim_splunk]

You definitely need to find out the reason and what makes your Splunkd busy.
In case you could not resolve the issue you may want to work around it by changing the timeouts (SPL-71676).

In version 6.x:

open/create $SPLUNK_HOME/etc/system/local/web.conf and increase the value.

splunkdConnectionTimeout = (default 30 secs)

Old versions:

Hard coded in $SPLUNK_HOME/lib/python2.7/site-packages/splunk/rest/__init__.py
Locate the below and increase to a higher - this would not survive any upgrades.

SPLUNKD_CONNECTION_TIMEOUT = 30

[AndrewEvelopers]

I have setup page in my app. Submitting form takes a lot of time and in the end it push the error
Splunkd daemon is not responding: ('The read operation timed out',).
The web-service.log says the error is coming from a REST-request from splunkweb to splunkd and failed by the timeout. Any ideas how to fix it?

[glitchcowboy · Accepted solution]

I had a log that generated more than 60,000 alert emails to me (badly configured alert on my part). Regardless, this nearly killed splunkd and I was getting this message as well. Not sure if this is an approved solution, but as I'm in test/build status currently, I stopped splunk and deleted everything in /opt/splunk/var/splunk/dispatch and that seemed to clear it up. Obviously this won't likely work too well in a production environment.

-- I was unable to use Splunk Web since splunkd was so clogged up.

[jangid]

I have deleted some views from Splunk web and now its working. still confused what is the relation between views and search job and who is running these jobs while my browser was closed even I haven't created any scheduled job/view/alert?

[jangid]

do you know how to view & delete search job?

[jangid]

No, I haven't removed anything.

[Ayn]

Did you remove some of the old search jobs?

[jangid]

One more warning along with error message

Too many search jobs found in the dispatch directory (found=16788, warning level=2000). This could negatively impact Splunk's performance, consider removing some of the old search jobs