Alerting

Help with alert setup

Crhis
New Member

Hi there, I am new to Splunk, so the question could be silly.... We set up an alert to alert out the on-call team once the first log of the day with the keyword "down" is detected by Splunk. However, it is very chatty. I wonder if it is possible to make an alert like below. 1. If the daily scan finds multiple "down" message in the past 24 hours, it only considers the most recent "down" message. 2. And Splunk will search for the following 7 days if there are any "up" messages. 3. Splunk only considers the most recent "up" message and as long as the time stamp of the "up" message is more recent than the "down" message, Splunk doesn't alert. Otherwise, it alerts the on-call team. The most difficult parts for me are: 1. How to trigger another query if the daily schedules find the down message. 2. How to keep the query running for the following 7 days. Any help would be much appreciated. Thank you,

Labels (2)
0 Karma

somesoni2
Revered Legend

Depending upon how data is getting logged in your environment, you could take the latest record for a device/servers (whatever is sending events with "up" and "down" messages) and alert if latest status is down. That way if the down device/server comes back up, no alerts will be fired.

e.g.

index=foo sourcetype=bar status=up OR status=down
| stats latest(status) as current_status by host_or_device
| where current_status="down"
0 Karma

Crhis
New Member

Good point. Thanks @somesoni2 

However, I don't want it to alert right away if the latest status is down. The plan is if the latest status is down, I want to keep an eye on it for the following 7 days. As long as the status is up, it doesn't alert. For example, let's say I will check the status at 12 am daily. The statuses are below. How can I tell Splunk the 1st up after two downs is for the 1st down but not for the second down. It only resets the 1st down, but keep scanning for the following 7 days for the second down.

 1st day2nd day3rd day4th day5th day6th day7th day8th day9th day
StatusDownDownUp DownDownDownDownDownDown
          
1st DownNo alertNo alertNo alert      
2nd Down No alertNo alertNo alertNo alertNo alertNo alertNo alertAlert

 

Thank you so much,

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...