Alerting

Alert Trigger Condition - Alert only if second string is not present

pierrem
Engager

Hi All, 

I'm currently trying to configure a alert to trigger when 2 events are NOT present in last 15min. 
In short if we have only Event1 but not Event2 then a alert should be triggered, if both events are present in last 15min then no alerts should be triggered. 

Use case, the alert is being configured to alert us when a VPN tunnel interface goes down and stays down for more than 15min, generally these VPN connections to terminate briefly but comes back up after a few seconds, hence we would like only alert if Event1 (down) took place in last 15min without Event2 (up) taking place. 

Event1 - Search query

index=firewall 10.10.10.10 Firewall_Name_XYZ=TEST123 AND "Lost Service"



Event2 - Search query 

index=firewall 10.10.10.10 Firewall_Name_XYZ=TEST123 AND (inbound "LAN-to-LAN" "created")



Search Query to show both events 

index=firewall 10.10.10.10 Firewall_Name_XYZ=TEST123 AND ("Lost Service" OR (inbound "LAN-to-LAN" "created"))



Any assistance will be greatly appreciated 🙂 

Labels (1)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust

You could try something like this

index=firewall 10.10.10.10 Firewall_Name_XYZ=TEST123 AND ("Lost Service" OR (inbound "LAN-to-LAN" "created"))
| rex "(?<event>Lost Service)"
| fillnull value="inbound created" event
| stats latest(event) as lastevent latest(_time) as lasttime
| where lastevent = "Lost Service" AND lasttime < now()-15*60

View solution in original post

pierrem
Engager

Thanks ITWhisperer 

It works like a charm, I just removed the lasttime statement as the alert is configured to run in a cron schedule searching last 15min 🙂 

Thanks for the quick assistance 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

You could try something like this

index=firewall 10.10.10.10 Firewall_Name_XYZ=TEST123 AND ("Lost Service" OR (inbound "LAN-to-LAN" "created"))
| rex "(?<event>Lost Service)"
| fillnull value="inbound created" event
| stats latest(event) as lastevent latest(_time) as lasttime
| where lastevent = "Lost Service" AND lasttime < now()-15*60
Get Updates on the Splunk Community!

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI!Discover how Splunk’s agentic AI ...

Splunk Enterprise Security 8.x: The Essential Upgrade for Threat Detection, ...

Watch On Demand the Tech Talk on November 6 at 11AM PT, and empower your SOC to reach new heights! Duration: ...

Splunk Observability as Code: From Zero to Dashboard

For the details on what Self-Service Observability and Observability as Code is, we have some awesome content ...