Idle log

[saotaigiri]

i need script in SPL to show when there is an idle forwarder or if a forwarder isn't forwarding

[richgalloway]

The Monitoring Console has an alert for that. Go to Settings->Monitoring Console->Settings->Alerts setup and look for "DMC Alert - Missing forwarders".

[saotaigiri]

It is greyed out, should i enable it?

[richgalloway · Accepted solution]

If you want to receive alerts about missing forwarders then, yes, you should enable it.

[saotaigiri]

Hello,

The server was turned off to test if the alert would work but t did not work. Please what can I do  to get an alert  where the forwarder is not getting any  data

[richgalloway]

"It did not work" doesn't help narrow the problem.
An alternative to the DMC alert is to create your own search for forwarder data. Save the search as an alert and have the alert trigger when the number of results is zero.

[saotaigiri]

Thanks for your reply, please could help to write the SPL query. I am not good at writing SPL queries.

[richgalloway]

I can help. What do you have so far?

[saotaigiri]

The query below is what I am using but it doesn't seem to work. Please can you look at it and if possible tweak to the correct one.

| rest /services/server/info | eval LastStartupTime=strftime(startuptime, "%Y/%m/%d %H:%M:%S")
| eval timenow=now()
| eval daysup = round((timenow - startuptime) / 86400,0)
| eval Uptime = tostring(daysup) + " Days"
| table splunk_server LastStartupTime Uptime

I am looking to get an alert when  a server or host meant to be feeding Splunk goes down. Thanks

[richgalloway]

That rest command will get you the status of your Splunk instances (indexers, search heads, etc), but not your forwarders

Try this query I found in the Splunk Security Essentials app. You should modify the index and sourcetype parameters to suit for environment.

| tstats prestats=t count(host) where index=* groupby host _time span=1d 
| tstats prestats=t append=t count where index=* sourcetype=* by host  _time span=1d 
| stats count(host) as all_logs count as win_logs by host _time 
| eval win_perc=round(100*(win_logs / all_logs), 2) 
| eventstats max(_time) as maxtime 
| stats count as num_data_samples avg(eval(if(_time<relative_time(maxtime, "-1d@d"), win_perc, null))) as avg sum(eval(if(_time<relative_time(maxtime, "-1d@d") AND win_perc=0, 1, null))) as past_instances_of_no_logs max(eval(if(_time>=relative_time(maxtime, "-1d@d"), win_perc, null))) as latest by host 
| where isnotnull(avg) AND num_data_samples>10 AND isnull(past_instances_of_no_logs) AND latest=0