Splunk ITSI

ITSI - Lag between KPI and Service Health Score

briannorthey
Engager

We're observing a lag between when the KPI data hits a threshold and when the KPI severity level changes color - and an even longer lag for when the Service changes color.

Example:
- Using Real Time for all Service Analyzers and Glass Tables
- KPI Search Schedule = 1 minute
- KPI changes color = 2 - 4 minutes
- Service changes color = 1 - 2 minutes after the KPI changes

The net result is a considerable lag on Glass Tables where only the Service Health Score is displayed.

Is there a way to change the configuration so there is less of a lag?

I understand a lag of up to 2 minutes (based on the KPI Search Schedule) but having a lag of up to 6 minutes on the Glass Table is not effective for our support teams.

yannK
Splunk Employee
Splunk Employee

This is not surprising.

  • data change on disk
  • the KPI run and update their summary values at best every minute - delay
  • the kpi is indexed
  • the service score is calculated based on the kpis values from the previous minute (as the current minute may not be indexed) + delay
  • the service score is indexed
  • if the service score has a dependency over another service, another minute of delay to wait for those dependent services healthscores + extra delay

also if you have indexing/forwarding slowness between the SH and the indexers, add some delay.
So it could take 2-3 minutes for the service score to flip.

0 Karma
Get Updates on the Splunk Community!

Uncovering Multi-Account Fraud with Splunk Banking Analytics

Last month, I met with a Senior Fraud Analyst at a nationally recognized bank to discuss their recent success ...

Secure Your Future: A Deep Dive into the Compliance and Security Enhancements for the ...

What has been announced?  In the blog, “Preparing your Splunk Environment for OpensSSL3,”we announced the ...

New This Month in Splunk Observability Cloud - Synthetic Monitoring updates, UI ...

This month, we’re delivering several platform, infrastructure, application and digital experience monitoring ...