Splunk Search

Send Alert if one event doesn't occur in 10 min

syjayaraj
Explorer

I have used transaction and non transaction method.
Non transaction method yields result but the results are not being sent as an alert.
Could some one help how to achieve without using stats earliest latest commands, probably using if condition.
Example:

Event1: hostname slot 1 down
Event2: hostname slot 1 up

If event 2 doesn't arive in 10 min , I need to get an alert ,
Give examples if event 2 also be in different source type

Tags (4)
0 Karma

woodcock
Esteemed Legend

Run this every minute for the last 11 minutes:

... | stats count count(eval(status="up")) AS ups max(_time) AS _time BY host slot
| where ups=0 and count>1 AND (now() - _time) > (10 * 60))

jkat54
SplunkTrust
SplunkTrust

“hostname slot 1 down” OR “hostname slot 1 up” _index_earliest=-15m@m _index_latest=-5m@m

Trigger condition is if number of results are less than 2, corn schedule is

 0,10,20,30,40,50 * * * *

Assumes you only get one “down” event and don’t have any fields extracted.

0 Karma

syjayaraj
Explorer

Index earliest I tried I believe;
Still I have to use stats earliest and latest right?

0 Karma

jkat54
SplunkTrust
SplunkTrust

No, this search is going to run every 10 minutes and account for indexing latency of up to 5 minutes. If there are less than two events found, then it will trigger the alert.

0 Karma

syjayaraj
Explorer

Not working in my case

0 Karma

Richfez
SplunkTrust
SplunkTrust

If you run the search it's based on as a regular search, does that return results? Would any of those returned results indicate the alert should have fired?

0 Karma

Richfez
SplunkTrust
SplunkTrust

Not really. At least in most cases I can think of the only time you might really need stats latest or earliest would be to do “more work” on the results afterwards. The use case you presented seems like it has simpler solutions, like the one that jkat54 provided.

I would use a cron of */10 * * * *

0 Karma

Richfez
SplunkTrust
SplunkTrust

Why is there the arbitrary constraint on not being able to use the stats earliest or latest commands? (It's not necessarily an issue in finding a solution, but I think many people who read this would like to know - have you had issues out of those before? Do you have a earliest/latest solution but someone's double-dared you to do it without?)

0 Karma

syjayaraj
Explorer

Dear Rich,
I have spent considerable amount of time to get an alert, till now no luck,
When run manually it shows the results but not as alert email(email never comes)

Transaction never works at all

Something splunk doesn't like,
I'll post exact search string

0 Karma

syjayaraj
Explorer

sourcetype=xyz ciscoLwappApIfDownNotify OR ciscoLwappApIfUpNotify OR bsnDot11StationAssociate | rex "CISCO-LWAPP-AP-MIB::cLApName.0

= STRING: (?<apname>\S+)" | rex "AIRESPACE-WIRELESS-MIB::bsnAPName.0 = STRING: \"(?<apname>\S+)?\"" | rex "CISCO-LWAPP-AP-MIB

::cLApDot11IfSlotId.0 = Gauge32: (?<radioslot>\S+)" | rex "AIRESPACE-WIRELESS-MIB::bsnStationAPIfSlotId.0 = INTEGER: (?<radioslot

>\S+)"| stats earliest(_time) as _time latest(_time) as time latest(snmpTrapOID_0) as action values(radioslot) as radioslot

values(cLApIfUpDownCause_0) as Cause values(cLApIfUpDownFailureCode_0) as failurecode values(cLApIfUpDownFailureType_0) as

failuretype values(cLApSysMacAddress_0) as apmac values(cLApAdminStatus_0) as adminstatus by apname | eval age=now()-time | where

age>300 AND action="OID: CISCO-LWAPP-AP-MIB::ciscoLwappApIfDownNotify"

I run this every 30 min for past 5 hrs, currently 2 results are shown but I don't receive as an alert

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...