Alerting

How to create a search to find if a switch interface is down, then trigger an alert only if the interface is not running

Mohanveera1
Explorer

Hi Splunk Experts,

I tried to create the search but can't be successful in it, I need a search, if in case the interface of an Cisco switch is down and doesn't came to up state within 5 minutes it should throw an alert. The logs are as below screenshot. I tried to write the query only for one scenario i.e if the switch status changed state to down i can get the alert but i can't merge the both message fields back to back.

Required Alert scenario:

when first log containing the message field with changed state to down appears and within 5 minutes the next log containing the message field with changed state to up doesn't appear then i must get an alert.

 

Mohanveera1_0-1667470543576.png

 

Thanks in advance.....

 

Labels (4)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Alerts can't wait to see what will happen in the next few minutes.  They only look at what happened in the past.  This problem typically is solved by looking for the most recent event from each device.  If it's a "down" event and it's been at least 5 minutes then trigger an alert.

index=cisco_switch sourcetype=syslog:cswitch (message="changed state to down" OR message="changed state to up")
```Get the most recent event for each device```
| dedup device_IP
```Keep only down devices at least 5 minutes old```
| where (message="change state to down" AND _time<=relative_time(now(), "-5m"))
| table _time device_IP

Set the alert to trigger is the number of results is not zero.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out &gt;&gt; As our brave ...