Monitoring Splunk

Impact of real-time distributed searches on CPU utilization?

gregbujak
Path Finder

What is the impact of running real-time searches across a Splunk cluster, both for the dedicated search head and the associated search peers?

The rule of thumb is that one search is one CPU core, does that also apply to distributed search (per peer)? So if your search peers have 8 cores, 8 real-time queries will consume all the cores, effectively grinding the cluster to a halt. Is my understanding correct?

Is there a functional difference between Splunk versions (I am particularly interested in 5.0.3)?

Thanks

gregbujak
Path Finder

I realized that joins in the RT query caused the acceleration to mis-behave. In the face of fast moving data, each instance of the same RT query would eat its share of the CPU (on my machine it was 20% per request). However, while expensive, in 6.0 this has been fixed. So now complex RT queries (with joins) are accelerated and work as expected.

The reality is that with fast moving data, instances of different RT queries will quickly consume all the cpu resources and grind the system to a halt.

ShaneNewman
Motivator

Remember that searches run on the Indexers, the compiling of data and post processing of the data happens on the search heads.

0 Karma

gregbujak
Path Finder

HI Alek, thanks for the response. Limits.conf does specify the max number real-time searches, but it doesn't explain how it works. So what would happen if we allow 8 rt queries at the same time?

0 Karma

aholzer
Motivator

As far as my understanding goes (will have to owe you the documentation) a real-time search won't consume an entire core. What I mean by this is that multiple real-time searches will share cores and therefore increase how many concurrent real-time searches you can run.

Look in the limits.conf for the real time search limits and the concurrent limits. I think they give an explanation of how it works there.

Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...