Alerting

Alerting when the log file is not updated for last 3 minutes

anandhalagaras1
Contributor

Hi Team,

We have multiple log files which will be regularly getting updated and the same will be ingested into Splunk. For example as mentioned in below query when i search the data for last 15 mintues i can see "n" number of events would be getting ingested for each and every minute and sometimes there would be multiple events in a single minute as well. So suppose if i search the specific (index=abc host="efg" machinedata OR xxxx-) query & doesnt have any events for the next 3 minutes then it should trigger an email alert to the concerned team.


Search query look like as below:
index=abc host="efg" machinedata OR xxxx-

Events return after search would be as below:
2020-06-19 05:15:53,083 INFO xxxx- splunk machinedata - Content Type : text/plain; charset=us-ascii xxxxxxx-xxxxx-xxxxx
2020-06-19 05:15:53,083 INFO xxxx- splunk machinedata - Body type: .net.lang.String xxxxx-xxxx-xxxxxxx
2020-06-19 05:15:52,881 DEBUG xxxx- splunk machinedata - [AccessMessage]: Matched: xxxx-xxxx-xxxx-xxxxx
2020-06-19 05:15:52,881 DEBUG xxxx- splunk machinedata - [abc (accept)]: Subject: [sample] abc def ijk XXXXXXXXXX

So kindly help with the query so that if there are no new events for last 3 minutes then it needs to trigger an email.

Labels (3)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @anandhalagaras1 ,

you have to use your search in alert, scheduled every 3 minutes (cron */3 * * * *) where the trigger condition is "Number of results is equal to 0".

Ciao.

Giuseppe

View solution in original post

0 Karma

rnowitzki
Builder

You could use your SPL as is and:

  • set the search time to either 3 minutes ago or (to allow some buffer) to something like from -4m@m to -1m@m 
  • in the alert use a trigger condition of "is equal to" 0
  • have it run with the cron schedule */3 * * * * and you should be good.

BR

edit:  @gcusello was quicker 🙂

--
Karma and/or Solution tagging appreciated.
0 Karma

anandhalagaras1
Contributor

thank you for your swift response.😀

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @anandhalagaras1 ,

you have to use your search in alert, scheduled every 3 minutes (cron */3 * * * *) where the trigger condition is "Number of results is equal to 0".

Ciao.

Giuseppe

0 Karma

anandhalagaras1
Contributor

thank you.

0 Karma
Get Updates on the Splunk Community!

Splunk + ThousandEyes: Correlate frontend, app, and network data to troubleshoot ...

 Are you tired of troubleshooting delays caused by siloed frontend, application, and network data? We've got a ...

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI!Discover how Splunk’s agentic AI ...

🔐 Trust at Every Hop: How mTLS in Splunk Enterprise 10.0 Makes Security Simpler

From Idea to Implementation: Why Splunk Built mTLS into Splunk Enterprise 10.0  mTLS wasn’t just a checkbox ...