Splunk IT Service Intelligence

How to backfill a ServiceHealthScore in ITSI?

skoelpin
SplunkTrust
SplunkTrust

I had to create a new service and backfilled the KPI's but the ServiceHealthScore does not backfill. How can I get this backfilled?

0 Karma
1 Solution

skoelpin
SplunkTrust
SplunkTrust

I ended up writing the SPL to calculate the ServiceHealthScore and backfilling it in the itsi_summary index. I confirmed it with the internal ITSI team and was told not to make the math behind it public

View solution in original post

0 Karma

skoelpin
SplunkTrust
SplunkTrust

I ended up writing the SPL to calculate the ServiceHealthScore and backfilling it in the itsi_summary index. I confirmed it with the internal ITSI team and was told not to make the math behind it public

0 Karma

PowerPacked
Builder

Hi @skoelpin

Good to hear, you got the solution.

Can you share how you were able to backfill a ServiceHealthScore KPI, as there is no option of backfill for this KPI in configure service page & Anywhere.

Not asking about maths & calculation.

Thanks

skoelpin
SplunkTrust
SplunkTrust

Yes. At a very high level, it will look like this. You must first backfill each KPI, in my case I did it into the regular summary index and joined it to the itsi_summary to keep it clean. This has 5 KPI's and 2 use adaptive thresholding and the other 3 use static thresholding. You can then use these values to display ServiceHealthScores in deep dives and glasstable views.

index=summary  KPI1 OR KPI2 OR KPI3 OR KPI4 OR KPI5
| timechart span=1m max(KPI1)  max(KPI2) max(KPI3) max(KPI4) max(KPI5)
| eval HourOfDay=strftime(_time, "%H")
| eval BucketMinuteOfHour=strftime(_time, "%M")
| eval DayOfWeek=strftime(_time, "%A")
| eval Sunday=if(like(DayOfWeek,"Sunday"),1,0)
| eval HourOfDay_2hourbar=if(HourOfDay%2==0,HourOfDay/2,null())
| filldown HourOfDay_2hourbar
| lookup xxxx_pdf.csv Sunday as Sunday HourOfDay_2hourbar as HourOfDay_2hourbar
| foreach max*
    [ eval "Z_<<FIELD>>" = ('<<FIELD>>' - 'avg(<<FIELD>>)' ) / 'stdev(<<FIELD>>)']
| rename "max(KPI1)" as KPI1
| eval KPI2 = case(KPI1==0,"Normal",KPI1<2,"Low",KPI1<4,"Medium",KPI1<6,"High",KPI1>=6,"Critical")
| rename "max(KPI2)" as KPI2
| eval KPI2_Level = case(KPI2==0,"Normal",KPI2<2,"Low",KPI2<4,"Medium",KPI2<6,"High",KPI2>=6,"Critical")
| rename "max(KPI3)" as KPI3
| eval KPI3_Level = case(KPI3==0,"Normal",KPI3>=1,"Critical")
| rename "max(KPI4)" as KPI4
| rename "max(KPI4)" as KPI4
| rename "*(*)" AS *_*
| eval KPI5 = case(Z_max_KPI4>=0,"Normal",Z_max_KPI4>=-2,"Low",Z_max_KPI4>=-3,"High",Z_max_KPI4<-3,"Critical")
| eval KPI5_Level  = case(Z_max_KPI5_Level<=0,"Normal",Z_max_KPI5<1,"Low",Z_max_KPI5<2,"Medium",Z_max_KPI5<3,"High",Z_max_KPI5>=3,"Critical")
| fields + *Level

<REDACTED>

| bin _time span=1m
| stats avg(Redacted1) AS xxxx avg(redacted2) AS xxxx avg(Redacted3) AS xxxx avg(Redacted4) AS xxxx avg(Redacted5) AS xxx  by _time
| eval ServiceHealthScore=(xxx+xxxx+xxx+xxx+xxx)/5
| timechart span=1m max(ServiceHealthScore) AS ServiceHealthScore
0 Karma

yannK
Splunk Employee
Splunk Employee

Currently ITSI (as of 3.1.X )
Only the KPI can be backfilled. (specifically non metrics KPI as of ITSI 3.1.X)
ITSI does not offer a way to backfill the service scores, because of the way the scores are being calculated.

There is a discussion to make it a feature in the future.

0 Karma

PowerPacked
Builder

Hi @skoelpin

Servicehealthscore is a KPI which is calculated based on the severity level and importance of all remaining KPI's existing in the same sercvice at that time.

Servicehealthscore is a score from 0 (critical) to 100(normal) --- calculation formulae = [weighted alert levels] / Σ [importance weights]

example:
KPINAME - ALERTSEVERITY - IMPORTNANCE
kpi1 - low - 10
kpi2 - high - 5

servicehealthscore = (70*10) + (30*5) / 10 + 5

Alert Severity and its scores are:
Normal - 100, Low- 70, Medium - 50, High - 30, Critical - 0.

so to say, when you did a backfill of all the kpis, they dont have a Threshold & Importnace set & servicehealthscore can't be calculated at that time.

Thanks

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Yes, I know.. This is why I asked the question with a bounty. I want to know how to backfill the health score now that all the KPIs are backfilled

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...