Hello,
So we ran a health check on our Splunk Search Head (w/ Splunk ITSI) and indexers. Though the results were pretty straightforward, would still like to know your inputs, assessments, and if possible, recommendation on this.
Current Situation:
Our Splunk ITSI is experiencing some frequent N/A KPIs which in my assessment would be the skipped searches due to 1000+ KPIs were already created (that's 1000+ saved searches every 5 minutes) and some Forwarders are not following the interval on the inputs.conf (from Add-On Apps, enabled scripted inputs especially on Windows)
Architecture:
3 Indexers
2 Search Heads (not clustered)
Search Head Statuses:
Search Head 1 is for miscellaneous reporting
- 5-10 concurrent users and some are viewing dashboards with expensive searches - (some searches are using summary data, but most are raw data)
Search Head 2 is dedicated for Splunk ITSI
- with 1200+ KPIs defined (scheduled searches) and Jobs are averaging from 700-1000+
- occasional warnings on dispatch directory and as observed, if many users are using, Searches are queuing which is to be expected.
Health Checks result:
Search Head w/ ITSI
Indexers
Thoughts?
Much appreciated!
... View more