realtime search scaling

[splunkbacon]

I have read about some limits you can come across when doing realtime searches. When trying to scale this out should you be editing these configuration files and increasing resources on the searchhead or the indexers? It seems like the indexers are doing all of the real searching. What exactly does the searchhead actually do in terms of the under the hood technical part of a search? Does it exist only to host the web gui and distribute searches to indexers?

[MuS]

Hi splunkbacon,

Have a read here https://conf.splunk.com/files/2016/slides/it-seemed-like-a-good-idea-at-the-time-architectural-anti-... with special focus on slide 12 😉
But yes, the indexers are the real bottleneck because each real time search takes up one CPU.

To prevent current and future headaches, you should forget about real time searches completely and run your alerts over short time ranges on short intervals.

Hope that helps ...

cheers, MuS

[splunkbacon]

I'm not sure I understand after reading that what I was getting at.

There are some limits with searching in regards to how many cores you have, and how many searches per core etc that determine how many realtime searches you can have going on. I'm not sure if these limits apply to only the searchhead or the indexer or if you should be updating them on both. Are the indexers the real bottleneck for realtime searches? I run into a lot of issues with realtime searches not firing with no indication as to why even though the search returns results when looking at a timespan.

[HiroshiSatoh]

If you read this part of the manual you will know what you are doing.

https://docs.splunk.com/Documentation/Splunk/7.2.5/Search/Writebettersearches
->Command types and parallel processing

For example, sorting by a large amount of data consumes more resources on the search head.