Alerting

How to create an alert in Splunk for Event A only if Event B doesn't also exist in a 30 second interval?

Magnus_001
Explorer

Hello,

I have an application that logs a "appsvr disconnected" and a "appsvr connected" message in the app log.  I have created the extraction called connectionstatus to indicate as such.  I would like to create a Splunk alert to notify me only when the connectionstatus = "appsvr disconnected" appears in the log and it is not followed by a connectionstatus = "appsvr connected" in a 30 second window.  I am trying to cut down on the false positives, it seems fairly straight forward but I haven't been able come up with a search that would satisfy this condition.  Any help is greatly appreciated.  Thanks!

Conditions:

connectionstatus="appsvr disconnected" appears in the log by itself for 30 seconds or more - Alert.

 connectionstatus="appsvr disconnected" appears in the log but the connectionstatus="appsvr connected" also appears in a 30 second window - False positive, don't alert.

-Magnus

Labels (2)
0 Karma

to4kawa
Ultra Champion

sample:

|makeresults count=2
| streamstats count
| eval _time=if(count=2,relative_time(_time,"-1d@h"),relative_time(_time,"@h"))
| makecontinuous span=5s _time
| eval connectionstatus="appsvr ".mvindex(split("disconnected,connected",","),random() % 2)
| streamstats count
| where count % 2 = 1
| autoregress connectionstatus as p
| where p!=connectionstatus
| rename COMMENT as "this is sample. from here, the logic"
| streamstats count(eval(connectionstatus="appsvr disconnected")) as session
| streamstats range(_time) as duration by session
| stats list(eval(strftime(_time,"%FT%T"))) as times max(duration) as duration list(connectionstatus) as connectionstatus by session
| sort session
| where duration > 30
0 Karma

isoutamo
SplunkTrust
SplunkTrust

I would like to add "where (duration > 0 OR duration == 0) to @to4kawa answer to catch cases where there is only disconnected but not connected after that.  

r. Ismo

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...