Splunk Search

One search job, one thread?

nickcode
Explorer

My deployment is: 1 forwarder + 2 indexers + 1 search head.
The forwarder has forwarded 50GB(about 100,000,000 events) to the two indexers;
When I launch a search like "sourcetype=xxx" from search head, I find the search performance really quite disappointing. Only 10,000 events can be scanned per second! That's to say, it will take about 3 hours to finish scaning all the events!
Each of my indexers has 24 cpus. And each time I launch a search from search head, only one cpu in each indexer is running about 100% while others keep free. It seems that one search job only works on one thread. That's quite a waste of my indexer' computing ability!
Are there any ways to config splunk to solve this problem?

0 Karma

nickcode
Explorer

Hm..Search heads distribute search requests across multiple indexers.
U can refer to http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Scaleyourdeployment

0 Karma

linu1988
Champion

As per my knowledge the indexer does only indexing, the search head does the searching. The search performance depends on the search head's capacity. If the performance is slower from your expectations, please check the running jobs and scheduled searches in your search head. That should indicate the cause of your performance degradation. And also please check the role under which you are running the searches too.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

The number of events scanned per second for one search usually is depending on your I/O rather than the number or speed of the CPUs.

As for your search itself, as long as you don't define any more specific filters splunk has to load 50GB off the disks and shove them towards the search head... that's a lot of work, what are you trying to find with the search?

martin_mueller
SplunkTrust
SplunkTrust

If you specify other keys splunk will use those keys to narrow down what needs to be looked at in the first place, that's what indexing data is for.

As a simple example, looking for sourcetype=x error will only consider events that contain the word error from that sourcetype, potentially cutting the amount of data that needs to be read from disk by a huge factor.

Personally, I see no point in specifying a query that loads and lists 50GB of data, but then I don't know what you're trying to achieve...

0 Karma

nickcode
Explorer

And my I/O is not the bottleneck the iostat shows..

0 Karma

nickcode
Explorer

If I change the search to "sourcetype=xxx key1=value1 AND key2=value2", indexers will also scan all the events since splunk only indexes events by timestamp, isn't it?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...