Splunk Search

How to edit my alert to turn it off during a known maintenance window?

cj039165
New Member

Hello -

I have an alert setup that I need to not fire on Sundays between 2:45pm and 4:00pm. Below is what I have configured. I'm getting alerts during the maintenance window. I could use a second set of eye to help spot what I have set wrong.

index=hdx_was source="/hdx1/was70-32/AppServer/profiles/AppSrv01/logs/PRD3_XF*/SystemOut.log" "A connection failed but has been re-established" | eval wday=strftime(now(),"%a"),homi=strftime(now(),"%H%M"),dactivate=if(wday="Sun" AND homi>=1445 AND homi<=1600,"off",NULL)

Thanks

0 Karma

goodsellt
Contributor

You should modify the search so it does not match events which fall under the maint window using a combination of the date_wday and date_hour and if it's on specific days every month then date_mday.

Ex, a maint window every sunday from 00:00:00 to 02:00:00, you could use "date_wday!=sunday AND (date_hour!=0 OR date_hour!=1)"

Another idea would be to put your maintenance schedule into a lookup CSV, either as their epoch times or a "normal" timestamp you'll have to convert with strptime. Then just filter out events with times which match those ranges in the search.

0 Karma

somesoni2
Revered Legend

What is your alert condition? Do you alert on "number of events greater than 0" type or any custom condition?

0 Karma

cj039165
New Member

It's currently set to 'per result'

0 Karma

somesoni2
Revered Legend

I see you calculated dactivate variable but not using it to filter results. I would just below to your current search.

your current search | where isnull(deactivate)

So if the alert ran on maintenance window, dactivate field will have value "off" and the search will not return any results.

0 Karma
Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...