Is it possible to identify Disk latency on Splunk indexers?

[mallempatisreed]

hi All,

Recently we have seens bucket corruption issues in our cluster. Upon chekcing with support they concluded that the root cause might be disk latency issues. Is it possible to monitor the Disk Health parameters of indexer servers in Splunk ?

Thanks,
Sree

[nilbak1]

You can look for resource usage for every splunk servers under DMC in Resource Usage tab like
CPU usage
Memory Usage
Disk Usage
I/O operations
I/O bandwidth utilization

You can also look for resource_usage.log for _introspection index for a particular splunk server

[Jeremiah]

Take a look at the Distributed Management Console, it should help you detect problems on your indexers, including poor disk performance (located on the Resource dashboards). You can configure a single DMC to collect data from all of your indexers and search heads.

http://docs.splunk.com/Documentation/Splunk/7.1.2/DMC/DMCoverview