I've found a post here - but I'm a bit confused on how to implement this or if there is another method ?
I'm monitoring a status event in a log file and want to know if that status stops being indexed.
if you have only to know when events flow is stopped you can run a simple search (like index=my_index sourcetype=my_sourcetype earliest=-300s latest=now) and save it as an alert scheduling every five minutes (cron
*/5 * * * *).
In instead you have to monitor when you don't receive events from an host in a list you have to create a lookup with all your monitored hosts (e.g. perimeter.csv) and run something like this:
| metasearch index=_internal earliest=-300s latest=now | eval host=upper(host) | stats count by host | append [ | inputlookup perimeter.csv | eval host=upper(host) , count=0 | fields host count ] | stats sum(count) AS Total by host | where Total=0
and then save it as an alert scheduling every five minutes (cron
*/5 * * * *)
I tried this method - but it doesn't seem to work.
created a lookupfile from a search and it looks like:
created the search :
| metasearch index=_internal earliest=-1m latest=now
| eval host=upper(host)
| stats count by host
| append [ | inputlookup hosts_list.csv | eval host=upper(host) , count=0 | fields host count ]
| stats sum(count) AS Total by host
| where Total=0
get the following as a result - even though the hosts are generating events every few seconds
if i then add a nonsense host to the lookup file ..
i then get the following output even though theres never been an event for the host TREACLE
are you sure that hosts in your lookup are in _internal?
"127.0.0.1" and "www.destinations.com" don't seem to be hostnames, you can verify this with a simple search index=_internal host="127.0.0.1" OR host="www.destinations.com".
From the result of your search they aren't known hosts.
If you don't put
| where Total=0, you have all the events from your hosts.
Verify hostnames and put in your lookup an hostname that you're sure to find in _internal and see if you find events.
Create a scheduled search for the event that needs to be there. Have the alert trigger if the search returns zero results.
@Skins, you will have to add more details of the event you are trying to monitor. Field name and sample data.
Following is an example of heartbeat query in Splunk Documentation: