splunk searches to be multithreaded in a single box

[mzorzi]

Is it possible to configure splunk searches to be multithreaded in a single box, that is - make single splunk-search process use more than one CPU . Just by running few simultaneous searches, single-threaded splunk-search process isn't able to use all available CPUs

[gkanapathy · Accepted solution]

No it is not. I have to say, this is the first time I've heard someone have trouble with too many CPUs for search. How many CPUs does this server have? Note that two or three will often be consumed by indexing, and it is useful to have at least two more free to run scheduled or summary searches.

If your customer is running the types of searches that would benefit from additional CPU, i.e., high-density reporting searches, they would probably benefit a great deal from summary indexing, which essentially consumes (batch) CPU earlier in order to speed up (interactive) searches later. If they are not running this type of search, then more threads won't help anyway, as the searches will be limited by disk I/O latency rather than number of CPUs.

[splunkears]

Ditto.
I also note that, when I've 24CPUs..only 2 are busy and rest are idle. And, my search takes a long time. Any pointers on parallelizing splunk search.
Has anyone tried http://code.google.com/p/ppss/ with splunk search on the same host / box.?

[a212830]

Ditto. Searches should be multi-threaded...

[mbrunetto]

I have the same problem. My searches are far more CPU limited than IO. Let me use all my CPU, then I can worry about buying new IO hardware, but let me have the choice.

[zscgeek]

Same thing here @ Voxeo - Some of our searches are CPU bound (field extractions etc) when going over vast amounts of data - would be really nice to be able to take advantage of the multi-core systems we have

[silvermail]

Yes, I would be interested to know if multi-threading searches will be implemented in the future. Currently the CPU cores (we are talking about 32 cores) are sitting idle most of the time except for one core when a complex transaction search is running. We have used iostat to monitor the disk I/O and can observe that we are not limited by the Disk I/O latency.

We don't have many users running searches simultaneously too, and if Splunk can support multi-threaded searches it be a huge performance boost for us.

[oreoshake]

I would definitely be interested if this ever changes. I've seen CPUs (24 cores) idling most of the time and I figure we might as well use them for something.