I've found a post here - but I'm a bit confused on how to implement this or if there is another method ?
https://answers.splunk.com/answers/475724/how-to-create-a-search-that-will-trigger-an-alert-1.html
I'm monitoring a status event in a log file and want to know if that status stops being indexed.
gratzi
Hi Skins,
if you have only to know when events flow is stopped you can run a simple search (like index=my_index sourcetype=my_sourcetype earliest=-300s latest=now) and save it as an alert scheduling every five minutes (cron */5 * * * *
).
In instead you have to monitor when you don't receive events from an host in a list you have to create a lookup with all your monitored hosts (e.g. perimeter.csv) and run something like this:
| metasearch index=_internal earliest=-300s latest=now
| eval host=upper(host)
| stats count by host
| append [ | inputlookup perimeter.csv | eval host=upper(host) , count=0 | fields host count ]
| stats sum(count) AS Total by host
| where Total=0
and then save it as an alert scheduling every five minutes (cron */5 * * * *
)
Bye.
Giuseppe
I tried this method - but it doesn't seem to work.
created a lookupfile from a search and it looks like:
host
"127.0.0.1"
"www.destinations.com"
created the search :
| metasearch index=_internal earliest=-1m latest=now
| eval host=upper(host)
| stats count by host
| append [ | inputlookup hosts_list.csv | eval host=upper(host) , count=0 | fields host count ]
| stats sum(count) AS Total by host
| where Total=0
get the following as a result - even though the hosts are generating events every few seconds
host Total
127.0.0.1 0
WWW.DESTINATIONS.COM 0
if i then add a nonsense host to the lookup file ..
host
"127.0.0.1"
"www.destinations.com"
"TREACLE"
i then get the following output even though theres never been an event for the host TREACLE
host Total
127.0.0.1 0
TREACLE 0
WWW.DESTINATIONS.COM 0
Hi Skins,
are you sure that hosts in your lookup are in _internal?
"127.0.0.1" and "www.destinations.com" don't seem to be hostnames, you can verify this with a simple search index=_internal host="127.0.0.1" OR host="www.destinations.com".
From the result of your search they aren't known hosts.
If you don't put | where Total=0
, you have all the events from your hosts.
Verify hostnames and put in your lookup an hostname that you're sure to find in _internal and see if you find events.
Bye.
Giuseppe
Create a scheduled search for the event that needs to be there. Have the alert trigger if the search returns zero results.
This is the way to go. Click Save As
--> Alert
and choose equal to No Results. This will say, if no results are returned, then fire the alert
@Skins, you will have to add more details of the event you are trying to monitor. Field name and sample data.
Following is an example of heartbeat query in Splunk Documentation:
https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Addinfo#2._Determine_which_heart...