How to mute alerts during maintenance?

[Rakzskull]

Hi 

I intend to mute the alerts at a specified time during the maintenance window, and they should start up again once the maintenance window is to finish.

Can someone please guide me on how to achieve this?  because Splunk doesn't have a feature for scheduling maintenance.

Thanks 🙂

[ITWhisperer]

So, the question is how would you tell Splunk when maintenance is happening? Once you know this, you can craft a search for your alerts to only return triggering results outside the maintenance time.

For example, if your alert is triggered on there being no results from the search, make sure there are (dummy) results returned during the maintenance period, or, if your alert is triggered on there being a number of results from the search, make sure no results are returned during the maintenance period.

[Rakzskull]

@ITWhisperer 

Thanks for the reply. 

We already have N number of alerts set up to go out when the count crosses certain thresholds.

Having all the alerts configured this this way is not a fesilbe choice.

Is there an achievable Splunk solution I can use to schedule a downtime?

[ITWhisperer]

The solution described is achievable, although perhaps not desirable! It might take a bit of effort, but once done, it might give you the control you require.

Having said that, I don't think there is a master switch to stop all alerts, although perhaps you could write a script to send a ReST command to disable each of the alerts (I am not sure if that's even possible though). You would need another script to re-enable the alerts of course.

You still haven't answered the fundamental question of how you tell Splunk that there is a maintenance period.

For example, some people use a lookup file with dates and time of maintenance periods, and the alerts all check whether a maintenance period is currently active and take appropriate action.

Depending on your overall alerting architecture, you may be able to take maintenance periods into account further downstream, e.g. if your alerts are integrated with Service Now, you may be able to suppress alert actions during maintenance periods.