Splunk ITSI

Splunk ITSI N/A KPIs

lloydknight
Builder

So we have SPLUNK ITSI and the KPIs were using OS logs.

The forwarders were set to send logs every 3 minutes (interval=180) i.e.

[script://./bin/ps.sh]
interval = 180
sourcetype = ps
source = ps
index = os
disabled = 0
_TCP_ROUTING = forwarder

Sometimes the OS logs were being indexed after 6 to 10 minutes and this causes our KPIs to N/A (we set N/A to Critical) and produces unnecessary red KPIs.

Our KPIs are CPU and Memory.

I just want to know why is this happening and what are the possible causes behind it?

Much Appreciated!

1 Solution

woodcock
Esteemed Legend

Go to your Monitoring Console and run the Health Checks and check for "skipped searches". It usually happens because you are either overloading your Search Head with work or the search that you wrote takes so long that it is not done before the next search is scheduled to start so it skips the next run (if it did not, you would enter an endless growth of searches that cannot finish).

View solution in original post

woodcock
Esteemed Legend

Go to your Monitoring Console and run the Health Checks and check for "skipped searches". It usually happens because you are either overloading your Search Head with work or the search that you wrote takes so long that it is not done before the next search is scheduled to start so it skips the next run (if it did not, you would enter an endless growth of searches that cannot finish).

Get Updates on the Splunk Community!

ATTENTION!! We’re MOVING (not really)

Hey, all! In an effort to keep this Slack workspace secure and also to make our new members' experience easy, ...

Splunk Admins: Build a Smarter Stack with These Must-See .conf25 Sessions

  Whether you're running a complex Splunk deployment or just getting your bearings as a new admin, .conf25 ...

AppDynamics Summer Webinars

This summer, our mighty AppDynamics team is cooking up some delicious content on YouTube Live to satiate your ...