Does Splunk use more than 4 cores?

[dhaffner]

I’m hoping someone can help answer this.
We have seen and heard some bits and pieces about Splunk really only using up to 4 cores on a Linux machine. Is this true? What is the real limit?
Basically, is it worth getting an indexer with 16 cores, or even 24? We are getting ready to order 3 new DL580’s for our environment and have been given the option of 24x128 machines. Is that just crazy overkill for an indexer? Is there any documentation that directly addresses this? I haven’t found much of anything other than a couple things here on answers, which don’t say for sure.

Thanks!

[araitz]

That is likely crazy overhead. Given how you guys use Splunk, you would be better off going for 3 indexers with 8 cores rather than one indexer with 24 cores.

You can install multiple instances of Splunk on a machine, and to some extent will experience better individual search performance. However, you will pay for it in terms of additional management complexity, increased contention, and adding a huge single point of failure.

[tedder · Accepted solution]

We have Splunk split across a handful of 16-core servers. Searches are single-threaded, so the determining factor is the number of concurrent users/searches.

Our experience is that we are much more IO bound than CPU-bound.

[araitz]

It is easy to confuse cores with processes with threads, but they aren't equivalent. Each search is a separate multi-threaded process. There are certain parts of a search process that are not implemented to use threads, while other parts may leverage multiple threads. That said, it is a safe rule of thumb that one search will use around one core for sizing purposes.

[dhaffner]

watching a couple indexers each 16x64, we have 6 or 7 splunkd PIDs going and loads around 8. A bit of SWAP is being used, too. That is pretty much normal for us. So I'd assume it would be best to stay with the 16x64 and 100 GB/day. Thoughts?

[dhaffner]

searches each use a core, right? and each one is sent out to the distributed indexers, right? So how many cores does just indexing use? Is it the indexing that is single threaded?

[tedder]

we should chat offline- they are bound to one core when we watch 'top'.

[araitz]

Searches are not single threaded.

[dhaffner]

Also, if we put more than one instance of Splunk on a 16x64 machine, will they use different cores and be more effective? Or will it just bottleneck at the drive I/O and network?
Thanks!