topic Splunk 7 upgrade - "ERROR DispatchThread - Failed to read runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_***/runtime.csv does not exist" in Splunk Search

Splunk 7 upgrade - "ERROR DispatchThread - Failed to read runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_***/runtime.csv does not exist"

splunkyouverymu — Wed, 22 Aug 2018 15:44:11 GMT

Hi All,

We just upgraded to Splunk 7 and a subsearch started auto-finalizing after 9000s timeout. Running this search by itself takes ~220s.

Search.log shows a long list of (900s worth) entries of:

ERROR DispatchThread - Failed to read runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_tmp_###/runtime.csv does not exist

I've seen plenty of old answers like this one https://answers.splunk.com/answers/104690/error-dispatchthread-error-reading-runtime-settings-file-does-not-exist-splunk-6-0-upgraded.html being a known issue in Splunk 6 and that it should be supressed. Curious if others are seeing this in Splunk 7 and if there is a better explanation of what is happening and how to resolve it.

Re: Splunk 7 upgrade - "ERROR DispatchThread - Failed to read runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_***/runtime.csv does not exist"

deepashri_123 — Fri, 24 Aug 2018 06:16:18 GMT

Hey@splunkyouverymuch,

Refer this ans:
https://answers.splunk.com/answers/501897/what-does-search-error-dispatchthread-error-readin.html

Let me know if this helps!!

Re: Splunk 7 upgrade - "ERROR DispatchThread - Failed to read runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_***/runtime.csv does not exist"

splunkyouverymu — Mon, 27 Aug 2018 14:58:21 GMT

Unfortunately, that does not help me. I'd read through that Splunk Answer previously, and I don't really see a solution.

Is that saying that the error needs to be treated differently on Admin side? Is there a way to ignore it to allow the search to continue running. Does this error indicate something is wrong that could affect data accuracy?

This report runs every night to populate a summary index, and the timeout seems to happen every other day.

Re: Splunk 7 upgrade - "ERROR DispatchThread - Failed to read runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_***/runtime.csv does not exist"

splunkyouverymu — Mon, 27 Aug 2018 15:31:17 GMT

I came across this Documentation about dispatch directory and performance.

I will get with one of our admins and see if perhaps our directory is too cluttered. The fact that this timeout does not happen every day leads me to suspect something that changes daily such as the amount of files existing in this directory.

Re: Splunk 7 upgrade - "ERROR DispatchThread - Failed to read runtime settings: File :/opt/splunk/var/run/splunk/di

yoho — Sat, 20 Jun 2020 09:55:20 GMT

I happen to have several long running searches having the same error message. Have you found any clue regarding these messages ?