topic Re: Exclude Daily Maintenance Time Range from daily alert in Getting Data In

Exclude Daily Maintenance Time Range from daily alert

cbwillh — Thu, 27 Aug 2020 15:59:28 GMT

I have two alerts which send alert emails whenever a server on our loadbalancer changes status from UP to DOWN or vice versa.

its working great but due to a really cheesy program we are forced to use it requires manual reboots every day or it hangs up during work hours preventing employees from working.

we have the servers scheduled to reboot every day between 0200 and 0330 hours

unfortunately this causes a daily spam storm from the alert I have configured which sends alert emails for each instance of the servers that are rebooting changing from up to down and down to up again.

I have found a lot of other posts regarding excluding time ranges but none of them that I tried have worked for me.

is there a way to edit my alert search to EXCLUDE any events with timestamps between 0200 and 0400 hours EVERY DAY?

my search is below

sourcetype=syslog_nsxedge host="NSX-Edge03-0" server!="NULL"
| rex "(?<json>{.*})"
| spath input=json systemEvents{} output=systemEvents
| stats values(_time) as _time by systemEvents
| spath input=systemEvents
| fields - systemEvents
| eval _time=strptime(timestamp,"%s")
| search message=*DOWN
| sort - _time
| table _time,eventCode,metaData.server,metaData.listener,eventCode,message,moduleName,severity
| rename metaData.listener TO Site,metaData.server TO Server

Re: Exclude Daily Maintenance Time Range from daily alert

gcusello — Thu, 27 Aug 2020 16:16:59 GMT

Hi @cbwillh,

add to your main search the condition: NOT (date_hour=2 OR date_hour=3), in this way

sourcetype=syslog_nsxedge host="NSX-Edge03-0" server!="NULL" NOT (date_hour=2 OR date_hour=3) | ...

Ciao.

Giuseppe

Re: Exclude Daily Maintenance Time Range from daily alert

cbwillh — Thu, 27 Aug 2020 17:35:07 GMT

Hello Giuseppe

thanks so much for your help. unfortunately I still get events during the ) 0200 - 0400 times

changed my search to below

sourcetype=syslog_nsxedge host="NSX-Edge03-0" server!="NULL" NOT (time_hour=2 OR time_hour=3)
| rex "(?<json>{.*})"
| spath input=json systemEvents{} output=systemEvents
| stats values(_time) as _time by systemEvents
| spath input=systemEvents
| fields - systemEvents
| eval _time=strptime(timestamp,"%s")
| search message=*DOWN
|sort - _time
| table _time,eventCode,metaData.server,metaData.listener,eventCode,message,moduleName,severity
| rename metaData.listener TO Site,metaData.server TO Server

results of new search with your suggestion below (note the events at 0210?)

_time eventCode Server Site message moduleName severity

2020-08-27 07:13:27	30302	Server69	Carson_MDCM_Servers	A Loadbalancer Server Status is changed to DOWN	vShield Edge LoadBalancer	Informational
2020-08-27 02:10:14	30302	Server81	WT_MDCM_Servers	A Loadbalancer Server Status is changed to DOWN	vShield Edge LoadBalancer	Informational
2020-08-27 02:10:03	30302	Server62	WT_MDCM_Servers	A Loadbalancer Server Status is changed to DOWN	vShield Edge LoadBalancer	Informational
2020-08-27 01:08:59	30302	Server69	Carson_MDCM_Servers	A Loadbalancer Server Status is changed to DOWN	vShield Edge LoadBalancer	Informational
2020-08-26 20:41:08	30302	Server62	WT_MDCM_Servers	A Loadbalancer Server Status is changed to DOWN	vShield Edge LoadBalancer	Informational
2020-08-26 20:34:53	30302	Server69	Carson_MDCM_Servers	A Loadbalancer Server Status is changed to DOWN	vShield Edge LoadBalancer	Informational
2020-08-26 17:12:13	30302	Server69	Carson_MDCM_Servers	A Loadbalancer Server Status is changed to DOWN	vShield Edge LoadBalancer	Informational

Re: Exclude Daily Maintenance Time Range from daily alert

gcusello — Fri, 28 Aug 2020 06:47:35 GMT

Hi @cbwillh,

sorry, my little error: the field to use is date_hour, so try:

sourcetype=syslog_nsxedge host="NSX-Edge03-0" server!="NULL" NOT (date_hour=2 OR date_hour=3) | ...

One little hint: use always also the index in the searches, you'll have faster searches.

Ciao.

Giuseppe

Re: Exclude Daily Maintenance Time Range from daily alert

cbwillh — Fri, 28 Aug 2020 14:58:12 GMT

Hello Giuseppe

Thanks so much for your help.

I edited my search to add the index and more important to add your suggested command and it works a treat.

really appreciate your taking the time to help me get this sorted. have a great day!

Will