We have a search with some subsearches that runs for about 40 seconds.
"This search has completed and has returned 11 results by scanning 6.296 events in 42,58 seconds".
Total runtime of the search is 84 seconds, from 09:25:39.645 until 09:27:05.635.
The last line in search.log is this: 09-03-2018 09:27:05.635 INFO PipelineComponent - Process delayed by 84.962 seconds, perhaps system was suspended?
Further examination of search.log shows these lines:
09-03-2018 09:25:39.861 INFO DispatchThread - Error reading runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_tmp_1535959539.1/runtime.csv does not exist
and a number of these:
09:25:47.590 ERROR DispatchThread - Failed to read runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_subsearch_subsearch_subsearch_subsearch_tmp_1535959542.9_1535959545.20_1535959545.21_1535959546.23_1535959546.25/runtime.csv does not exist
In search.log a total of 276 of both the INFO and the ERROR mentioning runtime.csv in some directory is present.
We are running Splunk 7.1.2 on an SH cluster with 2 indexer clusters. All machines run Linux and have SSD's with plenty of free memory, no swapping, plenty of free diskspace and the dispatch directory has about 2500 entries. The directory names are not too long for Linux.
Any ideas what we as Splunk admins can do to speed up the search? Eliminating the subsearches might solve the problem but I would like to make sure this is not an "undocumented feature" or misconfiguration on the server side. Until last month we were running on 6.6.2 and this did not occur as far as we know.
