Alerting

How to create an alert in Splunk for Event A only if Event B doesn't also exist in a 30 second interval?

Magnus_001
Explorer

Hello,

I have an application that logs a "appsvr disconnected" and a "appsvr connected" message in the app log.  I have created the extraction called connectionstatus to indicate as such.  I would like to create a Splunk alert to notify me only when the connectionstatus = "appsvr disconnected" appears in the log and it is not followed by a connectionstatus = "appsvr connected" in a 30 second window.  I am trying to cut down on the false positives, it seems fairly straight forward but I haven't been able come up with a search that would satisfy this condition.  Any help is greatly appreciated.  Thanks!

Conditions:

connectionstatus="appsvr disconnected" appears in the log by itself for 30 seconds or more - Alert.

 connectionstatus="appsvr disconnected" appears in the log but the connectionstatus="appsvr connected" also appears in a 30 second window - False positive, don't alert.

-Magnus

Labels (2)
0 Karma

to4kawa
Ultra Champion

sample:

|makeresults count=2
| streamstats count
| eval _time=if(count=2,relative_time(_time,"-1d@h"),relative_time(_time,"@h"))
| makecontinuous span=5s _time
| eval connectionstatus="appsvr ".mvindex(split("disconnected,connected",","),random() % 2)
| streamstats count
| where count % 2 = 1
| autoregress connectionstatus as p
| where p!=connectionstatus
| rename COMMENT as "this is sample. from here, the logic"
| streamstats count(eval(connectionstatus="appsvr disconnected")) as session
| streamstats range(_time) as duration by session
| stats list(eval(strftime(_time,"%FT%T"))) as times max(duration) as duration list(connectionstatus) as connectionstatus by session
| sort session
| where duration > 30
0 Karma

isoutamo
SplunkTrust
SplunkTrust

I would like to add "where (duration > 0 OR duration == 0) to @to4kawa answer to catch cases where there is only disconnected but not connected after that.  

r. Ismo

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...