Splunk ITSI N/A KPIs

[lloydknight]

So we have SPLUNK ITSI and the KPIs were using OS logs.

The forwarders were set to send logs every 3 minutes (interval=180) i.e.

[script://./bin/ps.sh]
interval = 180
sourcetype = ps
source = ps
index = os
disabled = 0
_TCP_ROUTING = forwarder

Sometimes the OS logs were being indexed after 6 to 10 minutes and this causes our KPIs to N/A (we set N/A to Critical) and produces unnecessary red KPIs.

Our KPIs are CPU and Memory.

I just want to know why is this happening and what are the possible causes behind it?

Much Appreciated!

[woodcock · Accepted solution]

Go to your Monitoring Console and run the Health Checks and check for "skipped searches". It usually happens because you are either overloading your Search Head with work or the search that you wrote takes so long that it is not done before the next search is scheduled to start so it skips the next run (if it did not, you would enter an endless growth of searches that cannot finish).