ITSI Threshold based on trend

[nzambo_splunk]

I'd like to set a threshold in ITSI based on how a KPI is trending. In this case, count of events trending towards 75,000 events per hour. So let's say it's 15 minutes past the hour and we're at 20,000 events, and the trend is upwards to the point where we'd be over 75,000 events this hour, I'd like to have my KPI turn orange instead of green.

I assume the threshold indicator could look at the trend, but I think I'd need the hourly projected count and not just the current trend.

Does this seem plausible or possible? Is there a more elegant way to accomplish this?

Thanks

[martin_mueller]

Given that you've said your data is pretty evenly distributed, I'd just count over the last 60 minutes and call it a day, KISS and all that.

[appache]

In your case you could use adaptive threshold option in services OR you could set up custom threshold values and set the importance to it so that KPI and service analyzer will show up the colors you want.

[nzambo_splunk]

Sorry, this doesn't really help. I'm not sure how adaptive thresholding is beneficial here since all I'm looking for is what the likely event count would be an hour from now if the current trend holds. I'm exploring the options above to see if we can predict the event count and use that to color code our KPIs. I appreciate the answer and would be open to exploring the option more, I just don't understand your meaning.

[martin_mueller]

Ignoring ITSI for a second, you could extrapolate or predict your KPI in the search itself to reflect a full hour, or you could run over a floating hour.
I'm not quite sure what your data is and how it ebbs and flows, so I don't know what approach fits your case best without more info.

[nzambo_splunk]

Yes this makes sense. Thinking through this, would you suggest looking at a timechart and using predict with future_timespan to get to this conclusion? It seems like we could span out the timechart over 10 or 15 minutes and predict 3-4 time spans into the future to get a field appropriate to use as a KPI.

The data doesn't come in waves, rather more of a relatively flat line so time based adaptive thresholding is less relevant here I think.

[sundareshr]

I think for what you want to accomplish, trendline would be a better option. Something like this

...earliest=-1h@h  | timechart span=5m count | trendline sma3(count) as sma_count | where count>sma_count*1.5

[nzambo_splunk]

Curious why trendline > predict in this case?