topic Re: Send Alert if one event doesn't occur in 10 min in Splunk Search

Send Alert if one event doesn't occur in 10 min

syjayaraj — Sun, 26 Nov 2017 10:03:56 GMT

I have used transaction and non transaction method.
Non transaction method yields result but the results are not being sent as an alert.
Could some one help how to achieve without using stats earliest latest commands, probably using if condition.
Example:

Event1: hostname slot 1 down
Event2: hostname slot 1 up

If event 2 doesn't arive in 10 min , I need to get an alert ,
Give examples if event 2 also be in different source type

Re: Send Alert if one event doesn't occur in 10 min

Richfez — Sun, 26 Nov 2017 12:48:53 GMT

Why is there the arbitrary constraint on not being able to use the stats earliest or latest commands? (It's not necessarily an issue in finding a solution, but I think many people who read this would like to know - have you had issues out of those before? Do you have a earliest/latest solution but someone's double-dared you to do it without?)

Re: Send Alert if one event doesn't occur in 10 min

jkat54 — Tue, 29 Sep 2020 16:56:59 GMT

“hostname slot 1 down” OR “hostname slot 1 up” _index_earliest=-15m@m _index_latest=-5m@m

Trigger condition is if number of results are less than 2, corn schedule is

 0,10,20,30,40,50 * * * *

Assumes you only get one “down” event and don’t have any fields extracted.

Re: Send Alert if one event doesn't occur in 10 min

syjayaraj — Sun, 26 Nov 2017 14:37:56 GMT

Dear Rich,
I have spent considerable amount of time to get an alert, till now no luck,
When run manually it shows the results but not as alert email(email never comes)

Transaction never works at all

Something splunk doesn't like,
I'll post exact search string

Re: Send Alert if one event doesn't occur in 10 min

syjayaraj — Sun, 26 Nov 2017 15:07:01 GMT

sourcetype=xyz ciscoLwappApIfDownNotify OR ciscoLwappApIfUpNotify OR bsnDot11StationAssociate | rex "CISCO-LWAPP-AP-MIB::cLApName.0

= STRING: (?<apname>\S+)" | rex "AIRESPACE-WIRELESS-MIB::bsnAPName.0 = STRING: \"(?<apname>\S+)?\"" | rex "CISCO-LWAPP-AP-MIB

::cLApDot11IfSlotId.0 = Gauge32: (?<radioslot>\S+)" | rex "AIRESPACE-WIRELESS-MIB::bsnStationAPIfSlotId.0 = INTEGER: (?<radioslot

>\S+)"| stats earliest(_time) as _time latest(_time) as time latest(snmpTrapOID_0) as action values(radioslot) as radioslot

values(cLApIfUpDownCause_0) as Cause values(cLApIfUpDownFailureCode_0) as failurecode values(cLApIfUpDownFailureType_0) as

failuretype values(cLApSysMacAddress_0) as apmac values(cLApAdminStatus_0) as adminstatus by apname | eval age=now()-time | where

age>300 AND action="OID: CISCO-LWAPP-AP-MIB::ciscoLwappApIfDownNotify"

I run this every 30 min for past 5 hrs, currently 2 results are shown but I don't receive as an alert

Re: Send Alert if one event doesn't occur in 10 min

syjayaraj — Sun, 26 Nov 2017 15:09:02 GMT

Index earliest I tried I believe;
Still I have to use stats earliest and latest right?

Re: Send Alert if one event doesn't occur in 10 min

Richfez — Sun, 26 Nov 2017 15:38:28 GMT

Not really. At least in most cases I can think of the only time you might really need stats latest or earliest would be to do “more work” on the results afterwards. The use case you presented seems like it has simpler solutions, like the one that jkat54 provided.

I would use a cron of */10 * * * *

Re: Send Alert if one event doesn't occur in 10 min

jkat54 — Mon, 27 Nov 2017 00:19:40 GMT

No, this search is going to run every 10 minutes and account for indexing latency of up to 5 minutes. If there are less than two events found, then it will trigger the alert.

Re: Send Alert if one event doesn't occur in 10 min

syjayaraj — Mon, 27 Nov 2017 17:21:25 GMT

Not working in my case

Re: Send Alert if one event doesn't occur in 10 min

Richfez — Mon, 27 Nov 2017 17:23:49 GMT

If you run the search it's based on as a regular search, does that return results? Would any of those returned results indicate the alert should have fired?

Re: Send Alert if one event doesn't occur in 10 min

woodcock — Mon, 04 Dec 2017 03:00:00 GMT

Run this every minute for the last 11 minutes:

... | stats count count(eval(status="up")) AS ups max(_time) AS _time BY host slot
| where ups=0 and count>1 AND (now() - _time) > (10 * 60))