topic Impact of real-time distributed searches on CPU utilization? in Monitoring Splunk

Impact of real-time distributed searches on CPU utilization?

gregbujak — Tue, 25 Jun 2013 20:57:49 GMT

What is the impact of running real-time searches across a Splunk cluster, both for the dedicated search head and the associated search peers?

The rule of thumb is that one search is one CPU core, does that also apply to distributed search (per peer)? So if your search peers have 8 cores, 8 real-time queries will consume all the cores, effectively grinding the cluster to a halt. Is my understanding correct?

Is there a functional difference between Splunk versions (I am particularly interested in 5.0.3)?

Thanks

Re: Impact of real-time distributed searches on CPU utilization?

aholzer — Tue, 25 Jun 2013 21:03:52 GMT

As far as my understanding goes (will have to owe you the documentation) a real-time search won't consume an entire core. What I mean by this is that multiple real-time searches will share cores and therefore increase how many concurrent real-time searches you can run.

Look in the limits.conf for the real time search limits and the concurrent limits. I think they give an explanation of how it works there.

Re: Impact of real-time distributed searches on CPU utilization?

gregbujak — Tue, 25 Jun 2013 21:43:41 GMT

HI Alek, thanks for the response. Limits.conf does specify the max number real-time searches, but it doesn't explain how it works. So what would happen if we allow 8 rt queries at the same time?

Re: Impact of real-time distributed searches on CPU utilization?

gregbujak — Fri, 25 Oct 2013 20:56:07 GMT

I realized that joins in the RT query caused the acceleration to mis-behave. In the face of fast moving data, each instance of the same RT query would eat its share of the CPU (on my machine it was 20% per request). However, while expensive, in 6.0 this has been fixed. So now complex RT queries (with joins) are accelerated and work as expected.

The reality is that with fast moving data, instances of different RT queries will quickly consume all the cpu resources and grind the system to a halt.

Re: Impact of real-time distributed searches on CPU utilization?

ShaneNewman — Fri, 25 Oct 2013 23:09:18 GMT

Remember that searches run on the Indexers, the compiling of data and post processing of the data happens on the search heads.