Splunk Search

Process delayed - Could you suggest a way to speed up our search?


We have a search with some subsearches that runs for about 40 seconds.
"This search has completed and has returned 11 results by scanning 6.296 events in 42,58 seconds".

Total runtime of the search is 84 seconds, from 09:25:39.645 until 09:27:05.635.

The last line in search.log is this: 09-03-2018 09:27:05.635 INFO PipelineComponent - Process delayed by 84.962 seconds, perhaps system was suspended?

Further examination of search.log shows these lines:

09-03-2018 09:25:39.861 INFO  DispatchThread - Error reading runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_tmp_1535959539.1/runtime.csv does not exist

and a number of these:

09:25:47.590 ERROR DispatchThread - Failed to read runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_subsearch_subsearch_subsearch_subsearch_tmp_1535959542.9_1535959545.20_1535959545.21_1535959546.23_1535959546.25/runtime.csv does not exist

In search.log a total of 276 of both the INFO and the ERROR mentioning runtime.csv in some directory is present.

We are running Splunk 7.1.2 on an SH cluster with 2 indexer clusters. All machines run Linux and have SSD's with plenty of free memory, no swapping, plenty of free diskspace and the dispatch directory has about 2500 entries. The directory names are not too long for Linux.

Any ideas what we as Splunk admins can do to speed up the search? Eliminating the subsearches might solve the problem but I would like to make sure this is not an "undocumented feature" or misconfiguration on the server side. Until last month we were running on 6.6.2 and this did not occur as far as we know.

Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...