Hi there, I am new to Splunk, so the question could be silly.... We set up an alert to alert out the on-call team once the first log of the day with the keyword "down" is detected by Splunk. However, it is very chatty. I wonder if it is possible to make an alert like below. 1. If the daily scan finds multiple "down" message in the past 24 hours, it only considers the most recent "down" message. 2. And Splunk will search for the following 7 days if there are any "up" messages. 3. Splunk only considers the most recent "up" message and as long as the time stamp of the "up" message is more recent than the "down" message, Splunk doesn't alert. Otherwise, it alerts the on-call team. The most difficult parts for me are: 1. How to trigger another query if the daily schedules find the down message. 2. How to keep the query running for the following 7 days. Any help would be much appreciated. Thank you,
Depending upon how data is getting logged in your environment, you could take the latest record for a device/servers (whatever is sending events with "up" and "down" messages) and alert if latest status is down. That way if the down device/server comes back up, no alerts will be fired.
e.g.
index=foo sourcetype=bar status=up OR status=down
| stats latest(status) as current_status by host_or_device
| where current_status="down"
Good point. Thanks @somesoni2
However, I don't want it to alert right away if the latest status is down. The plan is if the latest status is down, I want to keep an eye on it for the following 7 days. As long as the status is up, it doesn't alert. For example, let's say I will check the status at 12 am daily. The statuses are below. How can I tell Splunk the 1st up after two downs is for the 1st down but not for the second down. It only resets the 1st down, but keep scanning for the following 7 days for the second down.
1st day | 2nd day | 3rd day | 4th day | 5th day | 6th day | 7th day | 8th day | 9th day | |
Status | Down | Down | Up | Down | Down | Down | Down | Down | Down |
1st Down | No alert | No alert | No alert | ||||||
2nd Down | No alert | No alert | No alert | No alert | No alert | No alert | No alert | Alert |
Thank you so much,