How to create Alert for monitoring splunk forwarder that events is not fetched within the last 5 minutes

[avni26]

Hello,
I have multiple scripts in each host which send availability,memory,space details of servers to splunk in every 5 minutes.
I want to create alert when there is no events within 5 minutes and sent the notification that forwarder is not working for "host".
Please help me with sample search.

I tried with below query, but its not working as per my requirement. Its not sending notification for host which has no events as for that host its showing " no result found".

index="idx" sourcetype=scripts host IN (*) earliest=-5m@m latest=now
| bin span=5m _time | dedup host
| stats count as "event_count" by _time host| where event_count=0

[woodcock]

This issue is not as clear-cut as you might imagine and has been solved dozens of ways. Everybody who uses splunk eventually gets around to asking this same question and solving it some how. Before you do anything else, I would look 3 of the best options.

1 Check out the Forwarding Monitoring features in the Monitoring Console (https://docs.splunk.com/Documentation/Splunk/latest/DMC/ForwardersDeployment)

2 Check out the Broken Hosts App for Splunk (https://splunkbase.splunk.com/app/3247/)

3 Check out the new functionality for this use-case in the soon-to-be released update of the Splunk Security Essentials app (https://splunkbase.splunk.com/app/3435/); maybe @dveuve_splunk will comment on this.

[gcusello]

Hi avni26,
you have to create a lookup called e.g. perimeter.csv containing all the host to monitor (the hostname must be in a field called e.g. host), then runa search like this:

| metasearch index=_internal
| eval host=upper(host)
| stats count BY host
| append [ | inputlookup perimeter.csv host | eval count=0, host=upper(host) | fields host, count ]
| stats sum(count) AS Total BY host
| where Total=0

In this way if you have results, there are some hosts that aren't sending logs in the examined period.
You can generate an alert with this search or insert it in a dashboard panel.

Bye.
Giuseppe

[richgalloway]

You're getting "No results found" either because your search finds no events for the stats command to count or because there are no missing forwarders (all event_count values are not zero).

This approach, however, is flawed because Splunk cannot search for something that is not there. A working search will compare a list of forwarders seen in the last 5 minutes with a list of all expected forwarders and report the expected forwarders not on the 'seen' list.

BTW, the Monitoring Console has a missing forwarder alert built-in. You can enable it or use it as a model.

[avni26]

@ richgalloway Thank you for your response.
Yes , but I can't enable missing forwarder alert built in E3. Wanted to get alert by query only.

Please suggest how to achieve when there is no events (splunk could not receive data) from xyz server for abc source.

Thanks in advance.

[richgalloway]

If you can't enable the built-in alert you can still use it as a model for your own query.