How to backfill a ServiceHealthScore in ITSI?

[skoelpin]

I had to create a new service and backfilled the KPI's but the ServiceHealthScore does not backfill. How can I get this backfilled?

[skoelpin · Accepted solution]

I ended up writing the SPL to calculate the ServiceHealthScore and backfilling it in the itsi_summary index. I confirmed it with the internal ITSI team and was told not to make the math behind it public

[PowerPacked]

Hi @skoelpin

Good to hear, you got the solution.

Can you share how you were able to backfill a ServiceHealthScore KPI, as there is no option of backfill for this KPI in configure service page & Anywhere.

Not asking about maths & calculation.

Thanks

[skoelpin]

Yes. At a very high level, it will look like this. You must first backfill each KPI, in my case I did it into the regular summary index and joined it to the itsi_summary to keep it clean. This has 5 KPI's and 2 use adaptive thresholding and the other 3 use static thresholding. You can then use these values to display ServiceHealthScores in deep dives and glasstable views.

index=summary  KPI1 OR KPI2 OR KPI3 OR KPI4 OR KPI5
| timechart span=1m max(KPI1)  max(KPI2) max(KPI3) max(KPI4) max(KPI5)
| eval HourOfDay=strftime(_time, "%H")
| eval BucketMinuteOfHour=strftime(_time, "%M")
| eval DayOfWeek=strftime(_time, "%A")
| eval Sunday=if(like(DayOfWeek,"Sunday"),1,0)
| eval HourOfDay_2hourbar=if(HourOfDay%2==0,HourOfDay/2,null())
| filldown HourOfDay_2hourbar
| lookup xxxx_pdf.csv Sunday as Sunday HourOfDay_2hourbar as HourOfDay_2hourbar
| foreach max*
    [ eval "Z_<<FIELD>>" = ('<<FIELD>>' - 'avg(<<FIELD>>)' ) / 'stdev(<<FIELD>>)']
| rename "max(KPI1)" as KPI1
| eval KPI2 = case(KPI1==0,"Normal",KPI1<2,"Low",KPI1<4,"Medium",KPI1<6,"High",KPI1>=6,"Critical")
| rename "max(KPI2)" as KPI2
| eval KPI2_Level = case(KPI2==0,"Normal",KPI2<2,"Low",KPI2<4,"Medium",KPI2<6,"High",KPI2>=6,"Critical")
| rename "max(KPI3)" as KPI3
| eval KPI3_Level = case(KPI3==0,"Normal",KPI3>=1,"Critical")
| rename "max(KPI4)" as KPI4
| rename "max(KPI4)" as KPI4
| rename "*(*)" AS *_*
| eval KPI5 = case(Z_max_KPI4>=0,"Normal",Z_max_KPI4>=-2,"Low",Z_max_KPI4>=-3,"High",Z_max_KPI4<-3,"Critical")
| eval KPI5_Level  = case(Z_max_KPI5_Level<=0,"Normal",Z_max_KPI5<1,"Low",Z_max_KPI5<2,"Medium",Z_max_KPI5<3,"High",Z_max_KPI5>=3,"Critical")
| fields + *Level

<REDACTED>

| bin _time span=1m
| stats avg(Redacted1) AS xxxx avg(redacted2) AS xxxx avg(Redacted3) AS xxxx avg(Redacted4) AS xxxx avg(Redacted5) AS xxx  by _time
| eval ServiceHealthScore=(xxx+xxxx+xxx+xxx+xxx)/5
| timechart span=1m max(ServiceHealthScore) AS ServiceHealthScore

[yannK]

Currently ITSI (as of 3.1.X )
Only the KPI can be backfilled. (specifically non metrics KPI as of ITSI 3.1.X)
ITSI does not offer a way to backfill the service scores, because of the way the scores are being calculated.

There is a discussion to make it a feature in the future.

[PowerPacked]

Hi @skoelpin

Servicehealthscore is a KPI which is calculated based on the severity level and importance of all remaining KPI's existing in the same sercvice at that time.

Servicehealthscore is a score from 0 (critical) to 100(normal) --- calculation formulae = [weighted alert levels] / Σ [importance weights]

example:
KPINAME - ALERTSEVERITY - IMPORTNANCE
kpi1 - low - 10
kpi2 - high - 5

servicehealthscore = (70*10) + (30*5) / 10 + 5

Alert Severity and its scores are:
Normal - 100, Low- 70, Medium - 50, High - 30, Critical - 0.

so to say, when you did a backfill of all the kpis, they dont have a Threshold & Importnace set & servicehealthscore can't be calculated at that time.

Thanks

[skoelpin]

Yes, I know.. This is why I asked the question with a bounty. I want to know how to backfill the health score now that all the KPIs are backfilled