Splunk Search
Highlighted

How to edit my alert to turn it off during a known maintenance window?

New Member

Hello -

I have an alert setup that I need to not fire on Sundays between 2:45pm and 4:00pm. Below is what I have configured. I'm getting alerts during the maintenance window. I could use a second set of eye to help spot what I have set wrong.

index=hdx_was source="/hdx1/was70-32/AppServer/profiles/AppSrv01/logs/PRD3_XF*/SystemOut.log" "A connection failed but has been re-established" | eval wday=strftime(now(),"%a"),homi=strftime(now(),"%H%M"),dactivate=if(wday="Sun" AND homi>=1445 AND homi<=1600,"off",NULL)

Thanks

0 Karma
Highlighted

Re: How to edit my alert to turn it off during a known maintenance window?

SplunkTrust
SplunkTrust

What is your alert condition? Do you alert on "number of events greater than 0" type or any custom condition?

0 Karma
Highlighted

Re: How to edit my alert to turn it off during a known maintenance window?

New Member

It's currently set to 'per result'

0 Karma
Highlighted

Re: How to edit my alert to turn it off during a known maintenance window?

SplunkTrust
SplunkTrust

I see you calculated dactivate variable but not using it to filter results. I would just below to your current search.

your current search | where isnull(deactivate)

So if the alert ran on maintenance window, dactivate field will have value "off" and the search will not return any results.

0 Karma
Highlighted

Re: How to edit my alert to turn it off during a known maintenance window?

Contributor

You should modify the search so it does not match events which fall under the maint window using a combination of the datewday and datehour and if it's on specific days every month then date_mday.

Ex, a maint window every sunday from 00:00:00 to 02:00:00, you could use "datewday!=sunday AND (datehour!=0 OR date_hour!=1)"

Another idea would be to put your maintenance schedule into a lookup CSV, either as their epoch times or a "normal" timestamp you'll have to convert with strptime. Then just filter out events with times which match those ranges in the search.

0 Karma