Splunk Search

Process delayed - Could you suggest a way to speed up our search?


We have a search with some subsearches that runs for about 40 seconds.
"This search has completed and has returned 11 results by scanning 6.296 events in 42,58 seconds".

Total runtime of the search is 84 seconds, from 09:25:39.645 until 09:27:05.635.

The last line in search.log is this: 09-03-2018 09:27:05.635 INFO PipelineComponent - Process delayed by 84.962 seconds, perhaps system was suspended?

Further examination of search.log shows these lines:

09-03-2018 09:25:39.861 INFO  DispatchThread - Error reading runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_tmp_1535959539.1/runtime.csv does not exist

and a number of these:

09:25:47.590 ERROR DispatchThread - Failed to read runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_subsearch_subsearch_subsearch_subsearch_tmp_1535959542.9_1535959545.20_1535959545.21_1535959546.23_1535959546.25/runtime.csv does not exist

In search.log a total of 276 of both the INFO and the ERROR mentioning runtime.csv in some directory is present.

We are running Splunk 7.1.2 on an SH cluster with 2 indexer clusters. All machines run Linux and have SSD's with plenty of free memory, no swapping, plenty of free diskspace and the dispatch directory has about 2500 entries. The directory names are not too long for Linux.

Any ideas what we as Splunk admins can do to speed up the search? Eliminating the subsearches might solve the problem but I would like to make sure this is not an "undocumented feature" or misconfiguration on the server side. Until last month we were running on 6.6.2 and this did not occur as far as we know.

Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...