Monitoring Splunk

Impact of real-time distributed searches on CPU utilization?

gregbujak
Path Finder

What is the impact of running real-time searches across a Splunk cluster, both for the dedicated search head and the associated search peers?

The rule of thumb is that one search is one CPU core, does that also apply to distributed search (per peer)? So if your search peers have 8 cores, 8 real-time queries will consume all the cores, effectively grinding the cluster to a halt. Is my understanding correct?

Is there a functional difference between Splunk versions (I am particularly interested in 5.0.3)?

Thanks

gregbujak
Path Finder

I realized that joins in the RT query caused the acceleration to mis-behave. In the face of fast moving data, each instance of the same RT query would eat its share of the CPU (on my machine it was 20% per request). However, while expensive, in 6.0 this has been fixed. So now complex RT queries (with joins) are accelerated and work as expected.

The reality is that with fast moving data, instances of different RT queries will quickly consume all the cpu resources and grind the system to a halt.

ShaneNewman
Motivator

Remember that searches run on the Indexers, the compiling of data and post processing of the data happens on the search heads.

0 Karma

gregbujak
Path Finder

HI Alek, thanks for the response. Limits.conf does specify the max number real-time searches, but it doesn't explain how it works. So what would happen if we allow 8 rt queries at the same time?

0 Karma

aholzer
Motivator

As far as my understanding goes (will have to owe you the documentation) a real-time search won't consume an entire core. What I mean by this is that multiple real-time searches will share cores and therefore increase how many concurrent real-time searches you can run.

Look in the limits.conf for the real time search limits and the concurrent limits. I think they give an explanation of how it works there.

Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!