Splunk IT Service Intelligence
Highlighted

Force to run adaptive thresholding update in ITSI on all services

Explorer

When I select adaptive thresholding in Service Definition, it mentions
"Adaptive Thresholding runs everyday around midnight and updates the thresholding for the KPI based on the settings below. Once updated, old thresholds cannot be recovered."
but it is not updating anything. After month I had to open every single service and manually had to Apply Adaptive Thresholding to get it updated.

How do I run this using scheduled search instead of opening every single KPI out of 500+ applying manually ?

Highlighted

Re: Force to run adaptive thresholding update in ITSI on all services

SplunkTrust
SplunkTrust

What version of ITSI are you running and how have you validated that they are not updating?

0 Karma
Highlighted

Re: Force to run adaptive thresholding update in ITSI on all services

Explorer

ITSI 3.0.0 on Splunk 7.0.4
I have validated that, by observing for 30 days. KPI always in RED.
When I updated manually it did immediatelly fit into the thresholds correctly turning green

0 Karma
Highlighted

Re: Force to run adaptive thresholding update in ITSI on all services

SplunkTrust
SplunkTrust

How are your thresholds set? How many time policies do you have?

Have you looked at the time policies Preview window to determine how they look? Perhaps your values are trending much lower than they have historically? Have you had major outliers in the past which are making them look low now?

0 Karma
Highlighted

Re: Force to run adaptive thresholding update in ITSI on all services

Engager

I have a support case in for this very issue. Will let you know if I get a response that gets this working properly.

Highlighted

Re: Force to run adaptive thresholding update in ITSI on all services

Engager

Any updates on the support case ? I m seeing the similar issue in our ITSi environment. After changing threshold levels in time policies, the new values doesn't kick in automatically. I have to go to each KPI and click on "apply adaptive Threshold" after each modification.

Thanks.

0 Karma
Highlighted

Re: Force to run adaptive thresholding update in ITSI on all services

Explorer

We are running ITSI 3.1.4. Look like we are running into the same problem.
160 services are synced with the same service template. I have a response time KPI I want to edit the threshold template for. Static template are no problem syncing. But I want to use one of my adaptive template. Seems to me that I manually need to “apply adaptive threshold” for every Service. I haven't tried to wait over night, and will try that. But there shuld be a way to force update.

0 Karma
Highlighted

Re: Force to run adaptive thresholding update in ITSI on all services

Path Finder

Have there been any updates to this issue?

We too have services synced to Service Templates and thresholding linked to a Threshold template that employs adaptive thresholding. I would have thought that the adaptive thresholding would be applied to each service by learning the trend on the entities filtered in to that service (and not have to manually set each service threshold) but it does not appear to be working that way.

When using a Service Template and Threshold Templates with adaptive thresholding, what is used for training and does it uniquely apply to each service employing those templates?

0 Karma
Highlighted

Re: Force to run adaptive thresholding update in ITSI on all services

SplunkTrust
SplunkTrust

Adaptive thresholding will NOT work on a per entity basis. It only works on the aggregate value

There is a workaround, but it involves building a custom anomaly detection system in core Splunk to cook your entity data than passing it to ITSI.

0 Karma
Highlighted

Re: Force to run adaptive thresholding update in ITSI on all services

Path Finder

@skoelpin Thanks for the response. I didn't mean individual thresholds for each entity, I meant for the service aggregate, is there a different set of thresholds that get generated per service.

We have services using templates (both service and threshold templates). It seems that the threshold template is generating some thresholds based on something (not sure what) and applying that to all services. I would have thought that each service for which the template was applied would get a different adaptive threshold set applied to it.

What am I missing?

Thanks!

0 Karma