Alerting

How to create Splunk alert for event not generated in last 10 minutes?

ravir_jbp
Explorer

 

Trying to setup alert for two scenarios as metioned below:

Scenario 1: to determine if the connection between Xyz and the abc service is healthy, check for the string “IEX API Call Successfully got agent schedules data”   This message occurs in batches roughly every 5 minutes.  Good threshold might be to alert if This message is not seen in >= 10 minutes. 


Scenario 2: Another item to check would be the connection between the service and the xyz host.  The String for that is “Schedule successfully posted to the provider API”.  The cadence for those messages is the same so an absence of > 10 minutes may be a good place to start. 

Below are the samnple splunk events. I would like to setup an alert if these keywords event does not appears in last 10 minutes then send e-mail alert. Please help.

3/21/22
4:44:13.000 AM	
2022-03-21 04:44:13 [pool-6-thread-2] INFO  c.i.e.i.s.c.i.AgentResourceServiceImpl - IEX API Call Successfully got agent schedules data.

3/21/22
4:44:13.000 AM	 
2022-03-21 04:44:13 [pool-6-thread-2] INFO  c.i.e.i.s.c.i.AgentResourceServiceImpl - IEX API Call Successfully got agent schedules data.

3/21/22
4:44:13.000 AM	
2022-03-21 04:44:13 [pool-6-thread-2] INFO  c.i.e.i.s.c.i.AgentResourceServiceImpl - IEX API Call Successfully got agent schedules data.

3/21/22
4:44:13.000 AM	
2022-03-21 04:44:13 [pool-6-thread-2] INFO  c.i.e.f.a.w.s.i.SchedulesServiceImpl - Schedule successfully posted to the provider Api.

 

Labels (2)
Tags (1)
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Create a couple of reports which look at events in the previous 10  minutes (or 5 minutes if appropriate), one which searches for the first string and the other searches for the second string. If you get no results, trigger the alert with the send email action.

0 Karma

gcusello
Esteemed Legend

Hi @ravir_jbp,

you have to crea simple search like the following:

index=your_index “IEX API Call Successfully got agent schedules data” earliest=-10m@m latest=@m

or 

index=your_index “Schedule successfully posted to the provider API” earliest=-10m@m latest=@m

then create an alert for each search that triggers when there isn't any result (results=0).

Check the delay of your events, and eventually use a different time frame e.g. 

earliest=-15m@m latest=-5m@m

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...