How to create Splunk alert for event not generated in last 10 minutes?

[ravir_jbp]

Trying to setup alert for two scenarios as metioned below:

Scenario 1: to determine if the connection between Xyz and the abc service is healthy, check for the string “IEX API Call Successfully got agent schedules data”   This message occurs in batches roughly every 5 minutes.  Good threshold might be to alert if This message is not seen in >= 10 minutes. 


Scenario 2: Another item to check would be the connection between the service and the xyz host.  The String for that is “Schedule successfully posted to the provider API”.  The cadence for those messages is the same so an absence of > 10 minutes may be a good place to start. 

Below are the samnple splunk events. I would like to setup an alert if these keywords event does not appears in last 10 minutes then send e-mail alert. Please help.

3/21/22
4:44:13.000 AM	
2022-03-21 04:44:13 [pool-6-thread-2] INFO  c.i.e.i.s.c.i.AgentResourceServiceImpl - IEX API Call Successfully got agent schedules data.

3/21/22
4:44:13.000 AM	 
2022-03-21 04:44:13 [pool-6-thread-2] INFO  c.i.e.i.s.c.i.AgentResourceServiceImpl - IEX API Call Successfully got agent schedules data.

3/21/22
4:44:13.000 AM	
2022-03-21 04:44:13 [pool-6-thread-2] INFO  c.i.e.i.s.c.i.AgentResourceServiceImpl - IEX API Call Successfully got agent schedules data.

3/21/22
4:44:13.000 AM	
2022-03-21 04:44:13 [pool-6-thread-2] INFO  c.i.e.f.a.w.s.i.SchedulesServiceImpl - Schedule successfully posted to the provider Api.

[ITWhisperer]

Create a couple of reports which look at events in the previous 10  minutes (or 5 minutes if appropriate), one which searches for the first string and the other searches for the second string. If you get no results, trigger the alert with the send email action.

[gcusello]

Hi @ravir_jbp,

you have to crea simple search like the following:

index=your_index “IEX API Call Successfully got agent schedules data” earliest=-10m@m latest=@m

or 

index=your_index “Schedule successfully posted to the provider API” earliest=-10m@m latest=@m

then create an alert for each search that triggers when there isn't any result (results=0).

Check the delay of your events, and eventually use a different time frame e.g. 

earliest=-15m@m latest=-5m@m

Ciao.

Giuseppe