Getting Data In

Exclude Daily Maintenance Time Range from daily alert

cbwillh
Path Finder

I have two alerts which send alert emails whenever a server on our loadbalancer changes status from UP to DOWN or vice versa.

its working great but due to a really cheesy program we are forced to use it requires manual reboots every day or it hangs up during work hours preventing employees from working.

we have the servers scheduled to reboot every day between 0200 and 0330 hours

unfortunately this causes a daily spam storm from the alert I have configured which sends alert emails for each instance of the servers that are rebooting changing from up to down and down to up again.

I have found a lot of other posts regarding excluding time ranges but none of them that I tried have worked for me.

is there a way to edit my alert search to EXCLUDE any events with timestamps between 0200 and 0400 hours EVERY DAY?

my search is below

sourcetype=syslog_nsxedge host="NSX-Edge03-0" server!="NULL"
| rex "(?<json>{.*})"
| spath input=json systemEvents{} output=systemEvents
| stats values(_time) as _time by systemEvents
| spath input=systemEvents
| fields - systemEvents
| eval _time=strptime(timestamp,"%s")
| search message=*DOWN
| sort - _time
| table _time,eventCode,metaData.server,metaData.listener,eventCode,message,moduleName,severity
| rename metaData.listener TO Site,metaData.server TO Server

Labels (1)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @cbwillh,

sorry, my little error: the field to use is date_hour, so try:

sourcetype=syslog_nsxedge host="NSX-Edge03-0" server!="NULL" NOT (date_hour=2 OR date_hour=3)
| ...

One little hint: use always also the index in the searches, you'll have faster searches.

Ciao.

Giuseppe

 

View solution in original post

gcusello
SplunkTrust
SplunkTrust

Hi @cbwillh,

add to your main search the condition: NOT (date_hour=2 OR date_hour=3), in this way

 

sourcetype=syslog_nsxedge host="NSX-Edge03-0" server!="NULL" NOT (date_hour=2 OR date_hour=3)
| ...

 

 Ciao.

Giuseppe

cbwillh
Path Finder

Hello Giuseppe

thanks so much for your help. unfortunately I still get events during the ) 0200 - 0400 times 

changed my search to below

sourcetype=syslog_nsxedge host="NSX-Edge03-0" server!="NULL" NOT (time_hour=2 OR time_hour=3)
| rex "(?<json>{.*})"
| spath input=json systemEvents{} output=systemEvents
| stats values(_time) as _time by systemEvents
| spath input=systemEvents
| fields - systemEvents
| eval _time=strptime(timestamp,"%s")
| search message=*DOWN
|sort - _time
| table _time,eventCode,metaData.server,metaData.listener,eventCode,message,moduleName,severity
| rename metaData.listener TO Site,metaData.server TO Server

results of new search with your suggestion below (note the events at 0210?)

_time eventCode Server Site message moduleName severity

2020-08-27 07:13:2730302Server69Carson_MDCM_ServersA Loadbalancer Server Status is changed to DOWNvShield Edge LoadBalancerInformational
2020-08-27 02:10:1430302Server81WT_MDCM_ServersA Loadbalancer Server Status is changed to DOWNvShield Edge LoadBalancerInformational
2020-08-27 02:10:0330302Server62WT_MDCM_ServersA Loadbalancer Server Status is changed to DOWNvShield Edge LoadBalancerInformational
2020-08-27 01:08:5930302Server69Carson_MDCM_ServersA Loadbalancer Server Status is changed to DOWNvShield Edge LoadBalancerInformational
2020-08-26 20:41:0830302Server62WT_MDCM_ServersA Loadbalancer Server Status is changed to DOWNvShield Edge LoadBalancerInformational
2020-08-26 20:34:5330302Server69Carson_MDCM_ServersA Loadbalancer Server Status is changed to DOWNvShield Edge LoadBalancerInformational
2020-08-26 17:12:1330302Server69Carson_MDCM_ServersA Loadbalancer Server Status is changed to DOWNvShield Edge LoadBalancerInformational
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @cbwillh,

sorry, my little error: the field to use is date_hour, so try:

sourcetype=syslog_nsxedge host="NSX-Edge03-0" server!="NULL" NOT (date_hour=2 OR date_hour=3)
| ...

One little hint: use always also the index in the searches, you'll have faster searches.

Ciao.

Giuseppe

 

cbwillh
Path Finder

Hello Giuseppe

Thanks so much for your help.

I edited my search to add the index and more important to add your suggested command and it works a treat.

really appreciate your taking the time to help me get this sorted. have a great day!

Will

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...