Alerting

How to create Splunk alert for event not generated in last 10 minutes?

ravir_jbp
Explorer

 

Trying to setup alert for two scenarios as metioned below:

Scenario 1: to determine if the connection between Xyz and the abc service is healthy, check for the string “IEX API Call Successfully got agent schedules data”   This message occurs in batches roughly every 5 minutes.  Good threshold might be to alert if This message is not seen in >= 10 minutes. 


Scenario 2: Another item to check would be the connection between the service and the xyz host.  The String for that is “Schedule successfully posted to the provider API”.  The cadence for those messages is the same so an absence of > 10 minutes may be a good place to start. 

Below are the samnple splunk events. I would like to setup an alert if these keywords event does not appears in last 10 minutes then send e-mail alert. Please help.

3/21/22
4:44:13.000 AM	
2022-03-21 04:44:13 [pool-6-thread-2] INFO  c.i.e.i.s.c.i.AgentResourceServiceImpl - IEX API Call Successfully got agent schedules data.

3/21/22
4:44:13.000 AM	 
2022-03-21 04:44:13 [pool-6-thread-2] INFO  c.i.e.i.s.c.i.AgentResourceServiceImpl - IEX API Call Successfully got agent schedules data.

3/21/22
4:44:13.000 AM	
2022-03-21 04:44:13 [pool-6-thread-2] INFO  c.i.e.i.s.c.i.AgentResourceServiceImpl - IEX API Call Successfully got agent schedules data.

3/21/22
4:44:13.000 AM	
2022-03-21 04:44:13 [pool-6-thread-2] INFO  c.i.e.f.a.w.s.i.SchedulesServiceImpl - Schedule successfully posted to the provider Api.

 

Labels (2)
Tags (1)
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Create a couple of reports which look at events in the previous 10  minutes (or 5 minutes if appropriate), one which searches for the first string and the other searches for the second string. If you get no results, trigger the alert with the send email action.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @ravir_jbp,

you have to crea simple search like the following:

index=your_index “IEX API Call Successfully got agent schedules data” earliest=-10m@m latest=@m

or 

index=your_index “Schedule successfully posted to the provider API” earliest=-10m@m latest=@m

then create an alert for each search that triggers when there isn't any result (results=0).

Check the delay of your events, and eventually use a different time frame e.g. 

earliest=-15m@m latest=-5m@m

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...