Alerting
Highlighted

How to configure a Real Time Alert to ignore a scheduled maintenance window?

New Member

Greetings Splunkers,

I have a basic Real Time Alert (RTA) running that looks at all hosts for this message:

Message="The system has rebooted without cleanly shutting down first. This error could be caused if the system stopped responding, crashed, or lost power unexpectedly."

The RTA works as expected, however we reboot various servers every weekend for patches, which ends up triggering this alert. Is there anything I can do in the search parameters to ignore times from 12A-2A on weekends only? If not, is there another event code I can look at that truly is an unexpected shutdown alert?

Also on a side-note, is there a way to change my username? I didn't see the ability to in preferences.

0 Karma
Highlighted

Re: How to configure a Real Time Alert to ignore a scheduled maintenance window?

SplunkTrust
SplunkTrust

Precisely how is the RTA defined?

0 Karma
Highlighted

Re: How to configure a Real Time Alert to ignore a scheduled maintenance window?

New Member

Could you be more specific? Sorry

0 Karma
Highlighted

Re: How to configure a Real Time Alert to ignore a scheduled maintenance window?

SplunkTrust
SplunkTrust

see answer

0 Karma
Highlighted

Re: How to configure a Real Time Alert to ignore a scheduled maintenance window?

SplunkTrust
SplunkTrust

You have a search, you have scheduling options, throttling options, and so on. Your search has some time frame involved in it, and throws an alert if any records are returned in that time frame with the requested characteristics. To kill the alert, just make sure there are no records that come out of the end.

You could, for example, add a time element to the search using time to get the day and datehour to get the hour.

[your alert search ] 
| eval DayOfWeek=strftime(_time, "%A") 
| search DayOfWeek="Saturday" OR DayOfWeek="Sunday" OR date_hour>=2
| [any other formatting you had]
0 Karma
Highlighted

Re: How to configure a Real Time Alert to ignore a scheduled maintenance window?

Splunk Employee
Splunk Employee

To expand on this, you can have lookup which contains your defined maintenance windows. Then in the search, compare current alert time to the time window in that lookup. If it is within the time window, do not alert. If it is not in the time window, alert.....

Thats a very high level solution... Writing the search is a bit more complex..

0 Karma
Highlighted

Re: How to configure a Real Time Alert to ignore a scheduled maintenance window?

New Member

Ok thanks, trying to figure this one out. This is mine below with your help, though it doesn't seem to work.

host=* Message="The system has rebooted without cleanly shutting down first. This error could be caused if the system stopped responding, crashed, or lost power unexpectedly." | eval DayOfWeek=strftime(time, "%A")
| search DayOfWeek!="Saturday" AND date
hour!=0 OR datehour!=1 OR datehour!=2 AND DayOfWeek!="Sunday" AND datehour!=0 OR datehour!=1 OR date_hour!=2

0 Karma