Alerting

How do I get Splunk to generate an alert for multiple conditions?

Magnus_001
Explorer

Hello,

I am trying to monitor an application log and have Splunk generate an alert only when the  service_status = "disconnected" and  service_status="connected" entries are logged and the time between the two is greater than the span of 10 seconds OR if the Service_status = "disconnected" is the only entry being logged.   I've been experimenting with the transaction command but I am not getting the desired results.  Thanks in advance for any help with this.

Example log entries:

--- service is okay, do not generate an alert.---

9/2/2022 00:10:36.683   service_status = "disconnected"

9/2/2022 00:10:38.236  service_status="connected"

 

--- service is down, generate an alert.---

9/2/2022 00:10:40.683   service_status = "disconnected"

9/2/2022 00:10:51.736  service_status="connected"

 

--- service is down,  service_status="connected" event is missing,  generate an alert.---

9/2/2022 01:15:15.603   service_status = "disconnected"

 

Labels (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Magnus_001,

you could use the transaction command but it's a very slow command, so I hint a different approach:

index=your_index (service_status="disconnected" OR service_status="connected")
| bin span=10s _time
| stats dc(service_status) AS service_status_count values(service_status) AS service_status BY _time
| where service_status_count=1 AND service_status="disconnected"

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Splunk + ThousandEyes: Correlate frontend, app, and network data to troubleshoot ...

 Are you tired of troubleshooting delays caused by siloed frontend, application, and network data? We've got a ...

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI!Discover how Splunk’s agentic AI ...

🔐 Trust at Every Hop: How mTLS in Splunk Enterprise 10.0 Makes Security Simpler

From Idea to Implementation: Why Splunk Built mTLS into Splunk Enterprise 10.0  mTLS wasn’t just a checkbox ...