topic Re: How to stop alerts from being generated during maintenance. in Alerting

How to stop alerts from being generated during maintenance?

Rakzskull — Fri, 30 Sep 2022 14:30:19 GMT

I've seen a few posts on the subject, but I'd like to know how we can disable the multiple alerts throughout the maintenance window.

For example, I'd like to disable alerts 1, 2, and 3 from Saturday 11:30 p.m. until Sunday 6:00 a.m.

Thank you in advance.

------------------------------------

reference alert query

index=ABC sourcetype=XYZ ("Internal System Error")
|stats count
|where count >=30

Re: How to stop alerts from being generated during maintenance.

gcusello — Fri, 30 Sep 2022 07:20:33 GMT

Hi @Rakzskull,

if they are few, the easiest way it to manually disable them during maintenence period.

If you want to disable all the alert and you haven't scheduled reports or dashboards, you could disable the eMail configuration, so the alerts are triggered but the emails aren't sent.

There's a more elegant way, but it requires a little bit of work:

create a lookup (called e.g. maintenance.csv) containing only one columns (e.g. maintenance) and only two values (yes/not),
in each alert add the condition maintenance=not.
In this way, modifying the value in the lookup you stop all the alerts.

This surely is an interesting new feature, I hint to add it to Splunk Ideas.

Ciao.

Giuseppe

Re: How to stop alerts from being generated during maintenance.

chaker — Fri, 30 Sep 2022 07:52:36 GMT

It could also be done via the REST API:
https://community.splunk.com/t5/Alerting/How-do-you-disable-enable-alerts-via-the-REST-API/m-p/441558

There is also a good suggestion here to group the alerts by app, then disable the app:

https://community.splunk.com/t5/Alerting/How-can-we-suppress-a-set-of-alerts/m-p/480144

Need to add more points to this idea: +4 from me 😁

https://ideas.splunk.com/ideas/PLECID-I-297

Re: How to stop alerts from being generated during maintenance.

Rakzskull — Fri, 30 Sep 2022 08:25:18 GMT

@gcusello
I'm a rookie, so I don't know much about creating lookup csv. If you could explain the detailed technique with steps, I'd appreciate it. 🙂

Re: How to stop alerts from being generated during maintenance.

gcusello — Fri, 30 Sep 2022 08:54:03 GMT

Hi @Rakzskull,

if you don't know how to create a lookup I hint to follow the Splunk Search Tutorial (https://docs.splunk.com/Documentation/Splunk/9.0.1/SearchTutorial/WelcometotheSearchTutorial)

Anyway, you have to:

go in [Settings -- Lookup -- Lookup table files -- Lookup table files] and create the lookup with one column and one row
go in [Settings -- Lookup -- Lookup table files -- Lookup definitions] and create a definition for the lookup

Ciao.

Giuseppe