Splunk Search

Process delayed - Could you suggest a way to speed up our search?


We have a search with some subsearches that runs for about 40 seconds.
"This search has completed and has returned 11 results by scanning 6.296 events in 42,58 seconds".

Total runtime of the search is 84 seconds, from 09:25:39.645 until 09:27:05.635.

The last line in search.log is this: 09-03-2018 09:27:05.635 INFO PipelineComponent - Process delayed by 84.962 seconds, perhaps system was suspended?

Further examination of search.log shows these lines:

09-03-2018 09:25:39.861 INFO  DispatchThread - Error reading runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_tmp_1535959539.1/runtime.csv does not exist

and a number of these:

09:25:47.590 ERROR DispatchThread - Failed to read runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_subsearch_subsearch_subsearch_subsearch_tmp_1535959542.9_1535959545.20_1535959545.21_1535959546.23_1535959546.25/runtime.csv does not exist

In search.log a total of 276 of both the INFO and the ERROR mentioning runtime.csv in some directory is present.

We are running Splunk 7.1.2 on an SH cluster with 2 indexer clusters. All machines run Linux and have SSD's with plenty of free memory, no swapping, plenty of free diskspace and the dispatch directory has about 2500 entries. The directory names are not too long for Linux.

Any ideas what we as Splunk admins can do to speed up the search? Eliminating the subsearches might solve the problem but I would like to make sure this is not an "undocumented feature" or misconfiguration on the server side. Until last month we were running on 6.6.2 and this did not occur as far as we know.

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...