Alerting

Alerting when the log file is not updated for last 3 minutes

anandhalagaras1
Communicator

Hi Team,

We have multiple log files which will be regularly getting updated and the same will be ingested into Splunk. For example as mentioned in below query when i search the data for last 15 mintues i can see "n" number of events would be getting ingested for each and every minute and sometimes there would be multiple events in a single minute as well. So suppose if i search the specific (index=abc host="efg" machinedata OR xxxx-) query & doesnt have any events for the next 3 minutes then it should trigger an email alert to the concerned team.


Search query look like as below:
index=abc host="efg" machinedata OR xxxx-

Events return after search would be as below:
2020-06-19 05:15:53,083 INFO xxxx- splunk machinedata - Content Type : text/plain; charset=us-ascii xxxxxxx-xxxxx-xxxxx
2020-06-19 05:15:53,083 INFO xxxx- splunk machinedata - Body type: .net.lang.String xxxxx-xxxx-xxxxxxx
2020-06-19 05:15:52,881 DEBUG xxxx- splunk machinedata - [AccessMessage]: Matched: xxxx-xxxx-xxxx-xxxxx
2020-06-19 05:15:52,881 DEBUG xxxx- splunk machinedata - [abc (accept)]: Subject: [sample] abc def ijk XXXXXXXXXX

So kindly help with the query so that if there are no new events for last 3 minutes then it needs to trigger an email.

Labels (4)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @anandhalagaras1 ,

you have to use your search in alert, scheduled every 3 minutes (cron */3 * * * *) where the trigger condition is "Number of results is equal to 0".

Ciao.

Giuseppe

View solution in original post

0 Karma

rnowitzki
Builder

You could use your SPL as is and:

  • set the search time to either 3 minutes ago or (to allow some buffer) to something like from -4m@m to -1m@m 
  • in the alert use a trigger condition of "is equal to" 0
  • have it run with the cron schedule */3 * * * * and you should be good.

BR

edit:  @gcusello was quicker 🙂

--
Karma and/or Solution tagging appreciated.
0 Karma

anandhalagaras1
Communicator

thank you for your swift response.😀

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @anandhalagaras1 ,

you have to use your search in alert, scheduled every 3 minutes (cron */3 * * * *) where the trigger condition is "Number of results is equal to 0".

Ciao.

Giuseppe

0 Karma

anandhalagaras1
Communicator

thank you.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...