Identifying bottlenecks on indexers and search head

[bruceclarke]

Hi all,

I've been tasked with detailing information about our Splunk indexer and search head machines that shows the need for increased machine performance. I know our instance of Splunk is "slow" multiple times throughout the day. Moreover, I know that the machines do not meet the recommended hardware requirements. Regardless, I need to display this information to the best of my ability.

In order to relay this information, I want to use Splunk on Splunk in order to show what happens to our indexers and search head when under heavy load. Does anyone have some good recommendations for how to objectively display this information using S.o.S.?

Thanks!

[grijhwani]

You don't mention which platform you are running on, but this sounds like a sysadmin/resource issue, not an application issue. If you are running on Linux or unix, I would recommending using the "sar" utilities to profile machine resource usage over time across your Splunk infrastructure. This will give you a starting view of the resources being used and should indicate where they are maxing out, be that processor, memory, or i/o. From that you can then work back to understanding the processes, and the causes of the resource exhaustion.