topic Re: Scheduled Search Alert Suppression during Maintenance Window in Reporting

Scheduled Search Alert Suppression during Maintenance Window

dlazo — Sat, 12 Mar 2011 02:00:13 GMT

Hello, Is there a way to categorize or group Searches to suppress email alerts during a scheduled time period without having to go to each Search?

Re: Scheduled Search Alert Suppression during Maintenance Window

netwrkr — Sat, 12 Mar 2011 02:58:22 GMT

Yeah, I don't think this would be too difficult. My thought is:

For all searches you do not want to send alerts during a given time period (maintenance, I assume), instead of having them send out email, have them run a script. That script could say check for the existence of a file in a given location. If that file exists, suppress email alerts. If that file does not exist, send email as normal.

Re: Scheduled Search Alert Suppression during Maintenance Window

dlazo — Sat, 12 Mar 2011 03:37:10 GMT

I think I follow but that means we would have to change each alert - During to change it and make it run script instead, correct?

Re: Scheduled Search Alert Suppression during Maintenance Window

netwrkr — Tue, 15 Mar 2011 01:38:37 GMT

If I understand your question correctly - yes, you would have to make a one time change to any 'searches' that you don't want alerting during your maintenance window to filter first through a script. Instead of those searches/alerts going directly to email, tell them to use the script which we discussed above. HTH.

Re: Scheduled Search Alert Suppression during Maintenance Window

jrodman — Fri, 18 Mar 2011 16:03:58 GMT

You could also use the custom condition alert search and have that look at your maintenance window to control supression. This assumes you can express the window in terms of a splunk search.

Re: Scheduled Search Alert Suppression during Maintenance Window

bandit — Thu, 08 Dec 2016 19:19:38 GMT

The simplest out of the box way I can think of to achieve this is to group all alerts for a set of hosts, environment, or application into a single Splunk app space. This app space wouldn't have anything but the alerts defined. You would simply disable/enable that app in Splunk to turn off/on all associated alerts.

Re: Scheduled Search Alert Suppression during Maintenance Window

markbarber21 — Tue, 29 Sep 2020 16:51:02 GMT

Splunk Cloud does not allow us to Enable/Disable an application without a support ticket. Our use case does not have a predefined window, but needs a control by a user.

We can accomplish this using a search Macro.

1) Include your macro at the end of each related search. Ex: "should_run"
2) To enable the search define the macro to be:

| noop

3) To disable the search define the macro to be something like:

some_really_invalid_key="this will never be found"

Re: Scheduled Search Alert Suppression during Maintenance Window

tvaniderstine — Mon, 15 Oct 2018 16:50:27 GMT

My solution, which works for environments up to medium size, is to create multple alerts,for example
switch alerts - all
switch alerts - ignore esxi ports
swtich alerts - ignore firewall ports
switch alerts - ignore hsm ports

Normally, 'switch alerts - all' is enabled, but when maintenance is going occur on esxi hosts, we disabled 'switch alerts - all' and enable 'switch alerts - ignore esxi ports'.

This allows a bit of control without writing a script to query your ticket tracking system, and using that data to query your cabling database, and then using that data to update a generic 'link state' alert because in my opinion, telling a NOC to ignore alerts is the worst thing you can do.