The server was turned off to test if the alert would work but t did not work. Please what can I do to get an alert where the forwarder is not getting any data
The query below is what I am using but it doesn't seem to work. Please can you look at it and if possible tweak to the correct one.
| rest /services/server/info | eval LastStartupTime=strftime(startuptime, "%Y/%m/%d %H:%M:%S")
| eval timenow=now()
| eval daysup = round((timenow - startuptime) / 86400,0)
| eval Uptime = tostring(daysup) + " Days"
| table splunk_server LastStartupTime Uptime
I am looking to get an alert when a server or host meant to be feeding Splunk goes down. Thanks
That rest command will get you the status of your Splunk instances (indexers, search heads, etc), but not your forwarders
Try this query I found in the Splunk Security Essentials app. You should modify the index and sourcetype parameters to suit for environment.
| tstats prestats=t count(host) where index=* groupby host _time span=1d | tstats prestats=t append=t count where index=* sourcetype=* by host _time span=1d | stats count(host) as all_logs count as win_logs by host _time | eval win_perc=round(100*(win_logs / all_logs), 2) | eventstats max(_time) as maxtime | stats count as num_data_samples avg(eval(if(_time<relative_time(maxtime, "-1d@d"), win_perc, null))) as avg sum(eval(if(_time<relative_time(maxtime, "-1d@d") AND win_perc=0, 1, null))) as past_instances_of_no_logs max(eval(if(_time>=relative_time(maxtime, "-1d@d"), win_perc, null))) as latest by host | where isnotnull(avg) AND num_data_samples>10 AND isnull(past_instances_of_no_logs) AND latest=0