Why are only 4 CPUs being used on my 8 CPU search head?

[daniel333]

All,

When I "inspect" a job, I see that "command.search.kv" is my longest part. Please correct me if I am wrong about this, but that is the search head itself running various key value pair extractions against the data it received from the indexers? e.g. that is a CPU bottle neck on the search head.

Under that assumption I am watching top. I see on my 8 CPU search head, CPU 4 jumps to about 50% usage at most. The rest hover below 1%.

It seems to me that the search head is for some reason not taking full advantage of the CPUs I have allocated. Is there a way to tweak this? A setting I am missing? Or are my assumptions off?

[s2_splunk]

What search mode are you running your searches with (verbose, smart, fast)?
A single search uses at most one core on the SH, so it can only ever take advantage of multiple cores to run more than one concurrent searches.

[daniel333]

So looking at my search performance. In smart mode and verbose. I am overwhelmingly seeing that CPU "seems"to the bottle neck for me. If I run searches in fast mode for example they are 5x - 20x faster on specific index.

right now, I am working ton reducing knowledge objects and playing with the Data Curator to improve things. But I really feel like there should be a way to fork a search to take full advantage of hardware not being utilized?

[MuS]

Hi daniel333, if you're on Splunk 6.3.x take a look here http://docs.splunk.com/Documentation/Splunk/6.3.2/Capacity/Parallelization#Batch_mode_search_paralle... and learn how to turn on search parallelisation

Hope this helps ...

cheers, MuS

[daniel333]

Thanks for replying. I really appreciate it.

I have played with that setting in our dev environment and I am not seeing any real change. So I didn't see a reason to promote that config to prod.

Perhaps I am misunderstanding benefit of that setting. But as I am reading it, this should allow the indexer to dump data back to the SH must faster. Which, is not really the bottle neck for us.

So this specific index has nearly 400 unique key value pairs in it's logs. So the extraction times are really killing us.. i believe. So trying ot get more CPUs to help with that.