Force to run adaptive thresholding update in ITSI on all services

[dolezelk]

When I select adaptive thresholding in Service Definition, it mentions
"Adaptive Thresholding runs everyday around midnight and updates the thresholding for the KPI based on the settings below. Once updated, old thresholds cannot be recovered."
but it is not updating anything. After month I had to open every single service and manually had to Apply Adaptive Thresholding to get it updated.

How do I run this using scheduled search instead of opening every single KPI out of 500+ applying manually ?

[marcoscala]

Hi All! Any update on this issue? I'm having the same problem with Splunk ITSI 4.9.2 on Cloud. Just opened a case with the Support.

Regards,

Marco Scala

[sail4lot]

@skoelpin Thanks for the response. I didn't mean we were looking for adaptive thresholding on the entities. What I meant was that the adapting thresholding that is turned on in the threshold template does not appear to be creating unique thresholds adaptations for each of the services (which use service templates). It seems like it is creating a set of thresholds once a day for some large data set then applying that to all services that have the threshold template attached. That doesn't seem right to me. What am I missing?

Thanks!

[sail4lot]

Have there been any updates to this issue?

We too have services synced to Service Templates and thresholding linked to a Threshold template that employs adaptive thresholding. I would have thought that the adaptive thresholding would be applied to each service by learning the trend on the entities filtered in to that service (and not have to manually set each service threshold) but it does not appear to be working that way.

When using a Service Template and Threshold Templates with adaptive thresholding, what is used for training and does it uniquely apply to each service employing those templates?

[skoelpin]

Adaptive thresholding will NOT work on a per entity basis. It only works on the aggregate value

There is a workaround, but it involves building a custom anomaly detection system in core Splunk to cook your entity data than passing it to ITSI.

[sail4lot]

@skoelpin Thanks for the response. I didn't mean individual thresholds for each entity, I meant for the service aggregate, is there a different set of thresholds that get generated per service.

We have services using templates (both service and threshold templates). It seems that the threshold template is generating some thresholds based on something (not sure what) and applying that to all services. I would have thought that each service for which the template was applied would get a different adaptive threshold set applied to it.

What am I missing?

Thanks!

[skoelpin]

Adaptive thresholding is applied on a per KPI basis. You're wanting to do adaptive thresholding on the service level? You could sum the KPI's aggregate severities to get the service severity

[taskar]

We are running ITSI 3.1.4. Look like we are running into the same problem.
160 services are synced with the same service template. I have a response time KPI I want to edit the threshold template for. Static template are no problem syncing. But I want to use one of my adaptive template. Seems to me that I manually need to “apply adaptive threshold” for every Service. I haven't tried to wait over night, and will try that. But there shuld be a way to force update.

[cmutt78_2]

I have a support case in for this very issue. Will let you know if I get a response that gets this working properly.

[pvarjani]

Any updates on the support case ? I m seeing the similar issue in our ITSi environment. After changing threshold levels in time policies, the new values doesn't kick in automatically. I have to go to each KPI and click on "apply adaptive Threshold" after each modification.

Thanks.

[skoelpin]

What version of ITSI are you running and how have you validated that they are not updating?

[dolezelk]

ITSI 3.0.0 on Splunk 7.0.4
I have validated that, by observing for 30 days. KPI always in RED.
When I updated manually it did immediatelly fit into the thresholds correctly turning green

[skoelpin]

How are your thresholds set? How many time policies do you have?

Have you looked at the time policies Preview window to determine how they look? Perhaps your values are trending much lower than they have historically? Have you had major outliers in the past which are making them look low now?

[wmuselle]

Hi,

have you had any response to your support case?

We have run into exactly the same issue, AT bands get recalculated every night but do not kick in until you explicitly go in each service and go click 'apply adaptive' thresholding and 'save'.

this results in kpi being in the wrong band, impact service health score and basically rendering the service tree into an non reality-reflecting state