Alerting

How to trigger an alert if status event is not indexed for 5 minutes?

Skins
Path Finder

I've found a post here - but I'm a bit confused on how to implement this or if there is another method ?

https://answers.splunk.com/answers/475724/how-to-create-a-search-that-will-trigger-an-alert-1.html

I'm monitoring a status event in a log file and want to know if that status stops being indexed.

gratzi

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi Skins,
if you have only to know when events flow is stopped you can run a simple search (like index=my_index sourcetype=my_sourcetype earliest=-300s latest=now) and save it as an alert scheduling every five minutes (cron */5 * * * *).

In instead you have to monitor when you don't receive events from an host in a list you have to create a lookup with all your monitored hosts (e.g. perimeter.csv) and run something like this:

| metasearch index=_internal earliest=-300s latest=now
| eval host=upper(host) 
| stats count by host
| append [ | inputlookup perimeter.csv | eval host=upper(host) , count=0 | fields host count ]
| stats sum(count) AS Total by host
| where Total=0

and then save it as an alert scheduling every five minutes (cron */5 * * * *)
Bye.
Giuseppe

Skins
Path Finder

I tried this method - but it doesn't seem to work.

created a lookupfile from a search and it looks like:

host
"127.0.0.1"
"www.destinations.com"

created the search :

| metasearch index=_internal earliest=-1m latest=now
| eval host=upper(host)
| stats count by host
| append [ | inputlookup hosts_list.csv | eval host=upper(host) , count=0 | fields host count ]
| stats sum(count) AS Total by host
| where Total=0

get the following as a result - even though the hosts are generating events every few seconds

host Total
127.0.0.1 0
WWW.DESTINATIONS.COM 0

if i then add a nonsense host to the lookup file ..

host
"127.0.0.1"
"www.destinations.com"
"TREACLE"

i then get the following output even though theres never been an event for the host TREACLE

host Total
127.0.0.1 0
TREACLE 0
WWW.DESTINATIONS.COM 0

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi Skins,
are you sure that hosts in your lookup are in _internal?
"127.0.0.1" and "www.destinations.com" don't seem to be hostnames, you can verify this with a simple search index=_internal host="127.0.0.1" OR host="www.destinations.com".
From the result of your search they aren't known hosts.
If you don't put | where Total=0, you have all the events from your hosts.
Verify hostnames and put in your lookup an hostname that you're sure to find in _internal and see if you find events.
Bye.
Giuseppe

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Create a scheduled search for the event that needs to be there. Have the alert trigger if the search returns zero results.

---
If this reply helps you, Karma would be appreciated.
0 Karma

skoelpin
SplunkTrust
SplunkTrust

This is the way to go. Click Save As --> Alert and choose equal to No Results. This will say, if no results are returned, then fire the alert

0 Karma

niketn
Legend

@Skins, you will have to add more details of the event you are trying to monitor. Field name and sample data.

Following is an example of heartbeat query in Splunk Documentation:
https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Addinfo#2._Determine_which_heart...

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud’s AI Assistant in Action Series: Analyzing and ...

This is the second post in our Splunk Observability Cloud’s AI Assistant in Action series, in which we look at ...

Elevate Your Organization with Splunk’s Next Platform Evolution

 Thursday, July 10, 2025  |  11AM PDT / 2PM EDT Whether you're managing complex deployments or looking to ...

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...