I have a single Splunk instance with very high CPU workload. My investigation shows a bunch of searches are consuming all my CPU cores.
I would like to build a dedicated search head host and a separate indexer host to balance the workload.
Is ad-hoc/scheduled search requiring high CPU is normally running on search head or indexer? I guess it's on search head, just want to confirm if it's correct.
An indexer serves two functions. First, it takes in events and indexes them. Second, it services searches related to those events. Specifically, whatever part of a search is both streaming and distributable is services on the search head.
A search head serves to control searches. Any part of a search that occurs AFTER the last streaming and distributable command is executed on the search head.
If you are currently using a single host to do everything, then the first thing to do is check to see whether the searches that are taking up the most time have been properly optimized.
Here is an analogy and discussion of how Splunk architecture works and how to optimize searches.
Of course, if it is all on the same box, mostly the parts about getting rid of all unnecessary events and all unnecessary fields up front are paramount.
When you do eventually separate your system into indexer(s) and search head, it becomes critical to pay attention to the "streaming/distributable" vs "other" dichotomy as well, to avoid transmission of redundant or excessive data and to limit stress on the search head. Please consider adding two indexers as opposed to one, however. If you have active users and a fair number of events -- and it seems like you may due to your cpu load -- then the benefit from the second indexer is likely to be out of proportion to its cost. Also, this gets you further down the road before you need your next architectural upgrade, and if you do the thinking in advance, I believe you may be able in the future to upscale from two to three much easier than you would from one to two.
It's a bit hard for me to identify the streaming/un-streaming and distributable/un-distributable search commands because there are a bunch of search commands on my Splunk instance and all of them are written by other teams, and the amount is growing rapidly.
Is there any simple way to figure out the real CPU workload of indexing part and search part? I have checked out
Settings>Monitoring Console>Resource Usuage>Resource Usuage:Instance>Maximum CPU Usage by Process Class, I can tell the indexer only uses up to 1 CPU and it's quite stable, but the search use up to 6 CPU.
Does it mean my new indexer host only need 1 CPU, and new search header may need 6 CPU?