Splunk Search

Process delayed - Could you suggest a way to speed up our search?

tonniea
Explorer

We have a search with some subsearches that runs for about 40 seconds.
"This search has completed and has returned 11 results by scanning 6.296 events in 42,58 seconds".

Total runtime of the search is 84 seconds, from 09:25:39.645 until 09:27:05.635.

The last line in search.log is this: 09-03-2018 09:27:05.635 INFO PipelineComponent - Process delayed by 84.962 seconds, perhaps system was suspended?

Further examination of search.log shows these lines:

09-03-2018 09:25:39.861 INFO  DispatchThread - Error reading runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_tmp_1535959539.1/runtime.csv does not exist

and a number of these:

09:25:47.590 ERROR DispatchThread - Failed to read runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_subsearch_subsearch_subsearch_subsearch_tmp_1535959542.9_1535959545.20_1535959545.21_1535959546.23_1535959546.25/runtime.csv does not exist

In search.log a total of 276 of both the INFO and the ERROR mentioning runtime.csv in some directory is present.

We are running Splunk 7.1.2 on an SH cluster with 2 indexer clusters. All machines run Linux and have SSD's with plenty of free memory, no swapping, plenty of free diskspace and the dispatch directory has about 2500 entries. The directory names are not too long for Linux.

Any ideas what we as Splunk admins can do to speed up the search? Eliminating the subsearches might solve the problem but I would like to make sure this is not an "undocumented feature" or misconfiguration on the server side. Until last month we were running on 6.6.2 and this did not occur as far as we know.

Get Updates on the Splunk Community!

Aligning Observability Costs with Business Value: Practical Strategies

 Join us for an engaging Tech Talk on Aligning Observability Costs with Business Value: Practical ...

Mastering Data Pipelines: Unlocking Value with Splunk

 In today's AI-driven world, organizations must balance the challenges of managing the explosion of data with ...

Splunk Up Your Game: Why It's Time to Embrace Python 3.9+ and OpenSSL 3.0

Did you know that for Splunk Enterprise 9.4, Python 3.9 is the default interpreter? This shift is not just a ...