Splunk 7 upgrade - "ERROR DispatchThread - Failed to read runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_***/runtime.csv does not exist"

[splunkyouverymu]

Hi All,

We just upgraded to Splunk 7 and a subsearch started auto-finalizing after 9000s timeout. Running this search by itself takes ~220s.

Search.log shows a long list of (900s worth) entries of:

ERROR DispatchThread - Failed to read runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_tmp_###/runtime.csv does not exist

I've seen plenty of old answers like this one https://answers.splunk.com/answers/104690/error-dispatchthread-error-reading-runtime-settings-file-d... being a known issue in Splunk 6 and that it should be supressed. Curious if others are seeing this in Splunk 7 and if there is a better explanation of what is happening and how to resolve it.

[splunkyouverymu]

I came across this Documentation about dispatch directory and performance.

I will get with one of our admins and see if perhaps our directory is too cluttered. The fact that this timeout does not happen every day leads me to suspect something that changes daily such as the amount of files existing in this directory.

[yoho]

I happen to have several long running searches having the same error message. Have you found any clue regarding these messages ?

[deepashri_123]

Hey@splunkyouverymuch,

Refer this ans:
https://answers.splunk.com/answers/501897/what-does-search-error-dispatchthread-error-readin.html

Let me know if this helps!!

[splunkyouverymu]

Unfortunately, that does not help me. I'd read through that Splunk Answer previously, and I don't really see a solution.

Is that saying that the error needs to be treated differently on Admin side? Is there a way to ignore it to allow the search to continue running. Does this error indicate something is wrong that could affect data accuracy?

This report runs every night to populate a summary index, and the timeout seems to happen every other day.