Splunk Search

How to edit my alert to turn it off during a known maintenance window?

cj039165
New Member

Hello -

I have an alert setup that I need to not fire on Sundays between 2:45pm and 4:00pm. Below is what I have configured. I'm getting alerts during the maintenance window. I could use a second set of eye to help spot what I have set wrong.

index=hdx_was source="/hdx1/was70-32/AppServer/profiles/AppSrv01/logs/PRD3_XF*/SystemOut.log" "A connection failed but has been re-established" | eval wday=strftime(now(),"%a"),homi=strftime(now(),"%H%M"),dactivate=if(wday="Sun" AND homi>=1445 AND homi<=1600,"off",NULL)

Thanks

0 Karma

goodsellt
Contributor

You should modify the search so it does not match events which fall under the maint window using a combination of the date_wday and date_hour and if it's on specific days every month then date_mday.

Ex, a maint window every sunday from 00:00:00 to 02:00:00, you could use "date_wday!=sunday AND (date_hour!=0 OR date_hour!=1)"

Another idea would be to put your maintenance schedule into a lookup CSV, either as their epoch times or a "normal" timestamp you'll have to convert with strptime. Then just filter out events with times which match those ranges in the search.

0 Karma

somesoni2
Revered Legend

What is your alert condition? Do you alert on "number of events greater than 0" type or any custom condition?

0 Karma

cj039165
New Member

It's currently set to 'per result'

0 Karma

somesoni2
Revered Legend

I see you calculated dactivate variable but not using it to filter results. I would just below to your current search.

your current search | where isnull(deactivate)

So if the alert ran on maintenance window, dactivate field will have value "off" and the search will not return any results.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...