topic Re: Investigating high IO on indexers in Monitoring Splunk

Investigating high IO on indexers

jarush — Fri, 19 Jun 2020 01:16:32 GMT

We've recently migrated from 12 indexers per site on a slower storage array to 24 indexers per site on much faster storage arrays. Since the move we have seen IO throughput on indexer luns peak at around 6 - 8 GB/s, per site - for anywhere between 5 and 30 minutes. When that happens we start getting throttled by the storage array and latency goes up (as expected). We'd like to dig into the queries that are running at this time and see if we can do something about them (delete them, rewrite them, add datamodels, etc).

It's pretty easy to query the _internal index for sourcetype=scheduler and look at runtimes, etc. However, that doesn't give us an indication of how many buckets or slices were required to be examined by the indexers in order to satisfy the search.

Does anyone have recommendations, example searches, etc, that we can use to dig into this?

Re: Investigating high IO on indexers

satyenshahusda — Thu, 12 Mar 2020 17:45:10 GMT

Curious what version Splunk are you running? We recently had I/O issues on 8.0.1. Started spontaneously a couple of weekends ago.
Restarting individual indexers resolved it.

Re: Investigating high IO on indexers

jarush — Thu, 12 Mar 2020 18:33:20 GMT

7.3.3 across the board

Re: Investigating high IO on indexers

ivanreis — Fri, 13 Mar 2020 04:04:22 GMT

please verify the array IOPS specification, according to splunk docs, each drive should have 200 average IOPS. The configuration of the disks have to be Disks (RAID) 1+0 fault tolerance scheme as the disk
Here is the document with more information -> https://docs.splunk.com/Documentation/Splunk/8.0.2/Capacity/Referencehardware#Disk_subsystem

Check this steps to troubleshoot the index performance issues
https://docs.splunk.com/Documentation/Splunk/8.0.2/Troubleshooting/Troubleshootindexingperformance

and

https://docs.splunk.com/Documentation/Splunk/8.0.2/Troubleshooting/Troubleshootingeventsindexingdelay

Re: Investigating high IO on indexers

gjanders — Wed, 30 Sep 2020 04:37:54 GMT

In Alerts for Splunk Admins or github there are a few dashboards:
troubleshooting_indexer_cpu.xml
troubleshooting_resource_usage_per_user.xml

Or report wise summary / metrics searches:
SearchHeadLevel - platform_stats.user_stats.introspection metrics populating search

Would give you similar info but its more designed to output to a metrics index for later use...

Re: Investigating high IO on indexers

jarush — Fri, 13 Mar 2020 11:55:56 GMT

We have the app installed - i'm not really seeing anything in there that would help me drill into searches that are consuming the most IO. Is there a particular one you are thinking of?

Re: Investigating high IO on indexers

gjanders — Sat, 14 Mar 2020 04:08:50 GMT

Total read mb for example in the mentioned dashboard is coming from the introspection logs and related to I/O. Although this will measure from searches not ingestion

Re: Investigating high IO on indexers

jarush — Wed, 30 Sep 2020 04:38:22 GMT

Thanks, this got me going in the right direction. This ended up being the Splunk CS Toolkit app in the DMC. There were two queries that were destroying our storage: Splunk_index_lookup_genator and sta_forwarder_inventory. Using the below query we found they were doing two orders of magnitude more IO than all other queries:
index=_introspection host=* source=/resource_usage.log component=PerProcess data.process_type="search"
| stats sum(data.read_mb) by data.search_props.app, data.search_props.label

Re: Investigating high IO on indexers

gjanders — Sat, 14 Mar 2020 21:59:43 GMT

Glad it helped