We are about to add additional indexers to our Splunk infrastructure. We believe this will help with some of our search performance issues and indexing throughput issues. We also have seen that our current indexers will have high CPU (min of 8 core) and memory (min of 64GB).
Does anyone out there have some good searches or dashboards for measuring performance of the indexers? Anything to indicate whether the new indexers are making a difference for us? We do use SoS for looking at the index pipeline and some other metrics.
Add the indexer as search peer, Run search which you feel as slow. It should also have the data which is required. See inspect and check the run duration. This is quite abstract way to check if it has any performance improvement.
Have a faster disk rather than Memory and big CPU. 64 GB memory is mostly unnecessary. If required have 16 Core CPU. Until and unless your disk is capable of handling the indexing and give output to your searches faster you won't see the performance increase..