Impact of real-time distributed searches on CPU utilization?

[gregbujak]

What is the impact of running real-time searches across a Splunk cluster, both for the dedicated search head and the associated search peers?

The rule of thumb is that one search is one CPU core, does that also apply to distributed search (per peer)? So if your search peers have 8 cores, 8 real-time queries will consume all the cores, effectively grinding the cluster to a halt. Is my understanding correct?

Is there a functional difference between Splunk versions (I am particularly interested in 5.0.3)?

Thanks

[gregbujak]

I realized that joins in the RT query caused the acceleration to mis-behave. In the face of fast moving data, each instance of the same RT query would eat its share of the CPU (on my machine it was 20% per request). However, while expensive, in 6.0 this has been fixed. So now complex RT queries (with joins) are accelerated and work as expected.

The reality is that with fast moving data, instances of different RT queries will quickly consume all the cpu resources and grind the system to a halt.

[ShaneNewman]

Remember that searches run on the Indexers, the compiling of data and post processing of the data happens on the search heads.

[gregbujak]

HI Alek, thanks for the response. Limits.conf does specify the max number real-time searches, but it doesn't explain how it works. So what would happen if we allow 8 rt queries at the same time?

[aholzer]

As far as my understanding goes (will have to owe you the documentation) a real-time search won't consume an entire core. What I mean by this is that multiple real-time searches will share cores and therefore increase how many concurrent real-time searches you can run.

Look in the limits.conf for the real time search limits and the concurrent limits. I think they give an explanation of how it works there.