Scheduled Search Alert Suppression during Maintenance Window

[dlazo]

Hello, Is there a way to categorize or group Searches to suppress email alerts during a scheduled time period without having to go to each Search?

[tvaniderstine]

My solution, which works for environments up to medium size, is to create multple alerts,for example
switch alerts - all
switch alerts - ignore esxi ports
swtich alerts - ignore firewall ports
switch alerts - ignore hsm ports

Normally, 'switch alerts - all' is enabled, but when maintenance is going occur on esxi hosts, we disabled 'switch alerts - all' and enable 'switch alerts - ignore esxi ports'.

This allows a bit of control without writing a script to query your ticket tracking system, and using that data to query your cabling database, and then using that data to update a generic 'link state' alert because in my opinion, telling a NOC to ignore alerts is the worst thing you can do.

[markbarber21]

Splunk Cloud does not allow us to Enable/Disable an application without a support ticket. Our use case does not have a predefined window, but needs a control by a user.

We can accomplish this using a search Macro.

1) Include your macro at the end of each related search. Ex: "should_run"
2) To enable the search define the macro to be:

| noop

3) To disable the search define the macro to be something like:

some_really_invalid_key="this will never be found"

[bandit]

The simplest out of the box way I can think of to achieve this is to group all alerts for a set of hosts, environment, or application into a single Splunk app space. This app space wouldn't have anything but the alerts defined. You would simply disable/enable that app in Splunk to turn off/on all associated alerts.

[jrodman]

You could also use the custom condition alert search and have that look at your maintenance window to control supression. This assumes you can express the window in terms of a splunk search.

[netwrkr]

If I understand your question correctly - yes, you would have to make a one time change to any 'searches' that you don't want alerting during your maintenance window to filter first through a script. Instead of those searches/alerts going directly to email, tell them to use the script which we discussed above. HTH.

[netwrkr]

Yeah, I don't think this would be too difficult. My thought is:

For all searches you do not want to send alerts during a given time period (maintenance, I assume), instead of having them send out email, have them run a script. That script could say check for the existence of a file in a given location. If that file exists, suppress email alerts. If that file does not exist, send email as normal.

[dlazo]

I think I follow but that means we would have to change each alert - During to change it and make it run script instead, correct?