I am new to ITSI Splunk. I have created KPI based on event count based on the errors. if event count is 0 means normal. But if events was not logged due to issue(Like : Forwarder system) then also getting 0 as an output and KPI is showing Green. How to handle this ?
We have solved this by working out what the minimum number of log messages we would expect to see in a minute and if the number of events drops below that then we know there is an issue. It is probably slightly easier for us because our app services are all pooled resources that are fed by a load distributor that is regularly doing health checks so we know that we would expect a minimum of 3 health checks in a minute. We call this KPI 'Service-Alive' and it has a weighting of 11 in each service.
Hope that helps.
Maybe you can use the feature in thresholds called "Treat Gaps in Data as"
There are options to Flag it Critical when there are gaps in your data.
In ITSI, goto Configure --> Services -->
Find you service name and choose your KPI.
Under Thresholds the option is available : Treat Gaps in Data as
I think you might need to include another KPI. Or some other way of determining you have no data. Perhaps a count of good events. Something that will show your no data situation.
Or you could change your KPI so it is a percentage of events that are good and adjust your threshold. Then if it was 0 you would know you either have no data or you have a major issue.