Does anyone have examples of how to use Splunk to monitor storage I/O latency?
This use case enables system administrators to identify underlying hardware issues when systems experience storage latency.
This use case is from the Splunk Essentials for Infrastructure Troubleshooting and Monitoring app. For more examples, see the Splunk Essentials for Infrastructure Troubleshooting and Monitoring on Splunkbase.
How to implement: Ingest operating system logs and metrics into Splunk Enterprise. Install the Splunk Add-on for Windows or *nix, and enable iostat
monitoring for your entire infrastructure. Enable the iostat.sh
scripted input in the Splunk Add-on for Windows or *nix. Find the Splunk Add-on for Windows and Splunk Add-on for Unix and Linux on Splunkbase.
Data check: This use case depends on operating system logs and operating system metrics.
Track the latency of storage IO using the Splunk Add-on for Windows or *nix to identify storage and disks that are experiencing higher than normal latency. Build dashboards and alerts to take timely action.
Use the following search:
index=* tag=oshost tag=performance tag=storage
| stats avg(latency) AS average_wait_ms BY host
Best practice: In searches, replace the asterisk in index=*
with the name of the index that contains the data. By default, Splunk stores data in the main
index. Therefore, index=*
becomes index=main
. Use the OR
operator to specify one or multiple indexes to search. For example, index=main OR index=security
. See About managing indexes and How indexing works in Splunk docs for details.
If no results appear, deploy the Add-ons to the search heads to access the knowledge objects necessary for simple searching. See About installing Splunk add-ons on Splunk Docs for assistance.
For more support, post a question to the Splunk Answers community.
This use case enables system administrators to identify underlying hardware issues when systems experience storage latency.
This use case is from the Splunk Essentials for Infrastructure Troubleshooting and Monitoring app. For more examples, see the Splunk Essentials for Infrastructure Troubleshooting and Monitoring on Splunkbase.
How to implement: Ingest operating system logs and metrics into Splunk Enterprise. Install the Splunk Add-on for Windows or *nix, and enable iostat
monitoring for your entire infrastructure. Enable the iostat.sh
scripted input in the Splunk Add-on for Windows or *nix. Find the Splunk Add-on for Windows and Splunk Add-on for Unix and Linux on Splunkbase.
Data check: This use case depends on operating system logs and operating system metrics.
Track the latency of storage IO using the Splunk Add-on for Windows or *nix to identify storage and disks that are experiencing higher than normal latency. Build dashboards and alerts to take timely action.
Use the following search:
index=* tag=oshost tag=performance tag=storage
| stats avg(latency) AS average_wait_ms BY host
Best practice: In searches, replace the asterisk in index=*
with the name of the index that contains the data. By default, Splunk stores data in the main
index. Therefore, index=*
becomes index=main
. Use the OR
operator to specify one or multiple indexes to search. For example, index=main OR index=security
. See About managing indexes and How indexing works in Splunk docs for details.
If no results appear, deploy the Add-ons to the search heads to access the knowledge objects necessary for simple searching. See About installing Splunk add-ons on Splunk Docs for assistance.
For more support, post a question to the Splunk Answers community.
Update: I added a related video.