Alerting

How to blackout the alerts

srisplunk12
Engager

Is there a way to blackout all the alerts in Splunk during the maintenance window all at one go in Splunk Cloud

Tags (2)
0 Karma

DalJeanis
Legend

You could create a blackout macro and add it once to every alert search code.

woodcock has posted a version of the code in several places -- (search for blackout and cron together and you'll find a bunch of them__ here's a few links...

His version intentionally generates errors when the search runs during a blackout, but I modified it to a version that just quietly returns no results here...

The basic method for this silenced version is to add a subsearch that figures out whether you are running in a blackout window, and if so, adds a ridiculous search term that will never be found. That makes it certain that there will be zero records returned, and it should be pretty quick in doing so.

 YOUR SEARCH HERE 
  [ | noop | stats count AS search 
    | eval search=if(((tonumber(strftime(now(),"%W"))%2)==1),"SomeLongStringHereWhichWillNeverBeFoundInBloomFilters","")] 
 | YOUR PROCESSING HERE

Note the preceding example happened to be for a cron that ran once per week, but which really only wanted to run on alternate weeks. Thus, week number odd/even test was the method of deciding blackouts. Replace ((tonumber(strftime(now(),"%W"))%2)==1) with a test that returns true() during your blackouts and false() at all other times.


You may need to develop a similar but inverse method for any alert which is designed to complain when there are zero records returned...such as for when a host stops reporting. The format of that will depend on the typical architecture of your alerts. If you need that, then please open a new question, post an example of your alert that you need to hush during blackouts, and the community will be able to help you with a version that does what you need.

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...