Help with alert setup

[Crhis]

Hi there, I am new to Splunk, so the question could be silly.... We set up an alert to alert out the on-call team once the first log of the day with the keyword "down" is detected by Splunk. However, it is very chatty. I wonder if it is possible to make an alert like below. 1. If the daily scan finds multiple "down" message in the past 24 hours, it only considers the most recent "down" message. 2. And Splunk will search for the following 7 days if there are any "up" messages. 3. Splunk only considers the most recent "up" message and as long as the time stamp of the "up" message is more recent than the "down" message, Splunk doesn't alert. Otherwise, it alerts the on-call team. The most difficult parts for me are: 1. How to trigger another query if the daily schedules find the down message. 2. How to keep the query running for the following 7 days. Any help would be much appreciated. Thank you,

[somesoni2]

Depending upon how data is getting logged in your environment, you could take the latest record for a device/servers (whatever is sending events with "up" and "down" messages) and alert if latest status is down. That way if the down device/server comes back up, no alerts will be fired.

e.g.

index=foo sourcetype=bar status=up OR status=down
| stats latest(status) as current_status by host_or_device
| where current_status="down"

[Crhis]

Good point. Thanks @somesoni2 

However, I don't want it to alert right away if the latest status is down. The plan is if the latest status is down, I want to keep an eye on it for the following 7 days. As long as the status is up, it doesn't alert. For example, let's say I will check the status at 12 am daily. The statuses are below. How can I tell Splunk the 1st up after two downs is for the 1st down but not for the second down. It only resets the 1st down, but keep scanning for the following 7 days for the second down.

	1st day	2nd day	3rd day	4th day	5th day	6th day	7th day	8th day	9th day
Status	Down	Down	Up	Down	Down	Down	Down	Down	Down
1st Down	No alert	No alert	No alert
2nd Down		No alert	No alert	No alert	No alert	No alert	No alert	No alert	Alert

Thank you so much,