Alerting

How to stop alerts from being generated during maintenance?

Rakzskull
Path Finder

I've seen a few posts on the subject, but I'd like to know how we can disable the multiple alerts throughout the maintenance window.

For example, I'd like to disable alerts 1, 2, and 3 from Saturday 11:30 p.m. until Sunday 6:00 a.m.

Thank you in advance.

------------------------------------

reference alert query

index=ABC sourcetype=XYZ ("Internal System Error")
|stats count
|where count >=30

Labels (3)
Tags (1)
0 Karma

Rakzskull
Path Finder

@gcusello 
I'm a rookie, so I don't know much about creating lookup csv. If you could explain the detailed technique with steps, I'd appreciate it. 🙂 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Rakzskull,

if you don't know how to create a lookup I hint to follow the Splunk Search Tutorial (https://docs.splunk.com/Documentation/Splunk/9.0.1/SearchTutorial/WelcometotheSearchTutorial)

Anyway, you have to:

  • go in [Settings -- Lookup -- Lookup table files -- Lookup table files] and create the lookup with one column and one row
  • go in [Settings -- Lookup -- Lookup table files -- Lookup definitions] and create a definition for the lookup

Ciao.

Giuseppe

 

0 Karma

chaker
Contributor

It could also be done via the REST API:
https://community.splunk.com/t5/Alerting/How-do-you-disable-enable-alerts-via-the-REST-API/m-p/44155...

There is also a good suggestion here to group the alerts by app, then disable the app:

https://community.splunk.com/t5/Alerting/How-can-we-suppress-a-set-of-alerts/m-p/480144

 

Need to add more points to this idea:  +4 from me 😁

https://ideas.splunk.com/ideas/PLECID-I-297

 

gcusello
SplunkTrust
SplunkTrust

Hi @Rakzskull,

if they are few, the easiest way it to manually disable them during maintenence period.

If you want to disable all the alert and you haven't scheduled reports or dashboards, you could disable the eMail configuration, so the alerts are triggered but the emails aren't sent.

There's a more elegant way, but it requires a little bit of work:

  • create a lookup (called e.g. maintenance.csv) containing only one columns (e.g. maintenance) and only two values (yes/not),
  • in each alert add the condition maintenance=not.
  • In this way, modifying the value in the lookup you stop all the alerts.

This surely is an interesting new feature, I hint to add it to Splunk Ideas.

Ciao.

Giuseppe

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...