I have two alerts which send alert emails whenever a server on our loadbalancer changes status from UP to DOWN or vice versa.
its working great but due to a really cheesy program we are forced to use it requires manual reboots every day or it hangs up during work hours preventing employees from working.
we have the servers scheduled to reboot every day between 0200 and 0330 hours
unfortunately this causes a daily spam storm from the alert I have configured which sends alert emails for each instance of the servers that are rebooting changing from up to down and down to up again.
I have found a lot of other posts regarding excluding time ranges but none of them that I tried have worked for me.
is there a way to edit my alert search to EXCLUDE any events with timestamps between 0200 and 0400 hours EVERY DAY?
my search is below
sourcetype=syslog_nsxedge host="NSX-Edge03-0" server!="NULL"
| rex "(?<json>{.*})"
| spath input=json systemEvents{} output=systemEvents
| stats values(_time) as _time by systemEvents
| spath input=systemEvents
| fields - systemEvents
| eval _time=strptime(timestamp,"%s")
| search message=*DOWN
| sort - _time
| table _time,eventCode,metaData.server,metaData.listener,eventCode,message,moduleName,severity
| rename metaData.listener TO Site,metaData.server TO Server
Hi @cbwillh,
sorry, my little error: the field to use is date_hour, so try:
sourcetype=syslog_nsxedge host="NSX-Edge03-0" server!="NULL" NOT (date_hour=2 OR date_hour=3)
| ...
One little hint: use always also the index in the searches, you'll have faster searches.
Ciao.
Giuseppe
Hi @cbwillh,
add to your main search the condition: NOT (date_hour=2 OR date_hour=3), in this way
sourcetype=syslog_nsxedge host="NSX-Edge03-0" server!="NULL" NOT (date_hour=2 OR date_hour=3)
| ...
Ciao.
Giuseppe
Hello Giuseppe
thanks so much for your help. unfortunately I still get events during the ) 0200 - 0400 times
changed my search to below
sourcetype=syslog_nsxedge host="NSX-Edge03-0" server!="NULL" NOT (time_hour=2 OR time_hour=3)
| rex "(?<json>{.*})"
| spath input=json systemEvents{} output=systemEvents
| stats values(_time) as _time by systemEvents
| spath input=systemEvents
| fields - systemEvents
| eval _time=strptime(timestamp,"%s")
| search message=*DOWN
|sort - _time
| table _time,eventCode,metaData.server,metaData.listener,eventCode,message,moduleName,severity
| rename metaData.listener TO Site,metaData.server TO Server
results of new search with your suggestion below (note the events at 0210?)
_time eventCode Server Site message moduleName severity
2020-08-27 07:13:27 | 30302 | Server69 | Carson_MDCM_Servers | A Loadbalancer Server Status is changed to DOWN | vShield Edge LoadBalancer | Informational |
2020-08-27 02:10:14 | 30302 | Server81 | WT_MDCM_Servers | A Loadbalancer Server Status is changed to DOWN | vShield Edge LoadBalancer | Informational |
2020-08-27 02:10:03 | 30302 | Server62 | WT_MDCM_Servers | A Loadbalancer Server Status is changed to DOWN | vShield Edge LoadBalancer | Informational |
2020-08-27 01:08:59 | 30302 | Server69 | Carson_MDCM_Servers | A Loadbalancer Server Status is changed to DOWN | vShield Edge LoadBalancer | Informational |
2020-08-26 20:41:08 | 30302 | Server62 | WT_MDCM_Servers | A Loadbalancer Server Status is changed to DOWN | vShield Edge LoadBalancer | Informational |
2020-08-26 20:34:53 | 30302 | Server69 | Carson_MDCM_Servers | A Loadbalancer Server Status is changed to DOWN | vShield Edge LoadBalancer | Informational |
2020-08-26 17:12:13 | 30302 | Server69 | Carson_MDCM_Servers | A Loadbalancer Server Status is changed to DOWN | vShield Edge LoadBalancer | Informational |
Hi @cbwillh,
sorry, my little error: the field to use is date_hour, so try:
sourcetype=syslog_nsxedge host="NSX-Edge03-0" server!="NULL" NOT (date_hour=2 OR date_hour=3)
| ...
One little hint: use always also the index in the searches, you'll have faster searches.
Ciao.
Giuseppe
Hello Giuseppe
Thanks so much for your help.
I edited my search to add the index and more important to add your suggested command and it works a treat.
really appreciate your taking the time to help me get this sorted. have a great day!
Will