topic Alert when a Splunk service is down in Alerting

Alert when a Splunk service is down

nawazns5038 — Fri, 27 Apr 2018 00:13:23 GMT

Hi,

Suppose we have 10 heavy forwarders and want to get alerted if any one of them goes down.
How do we form an alert query.

index=_internal source=*splunkd.log* may work for for a single server, how to extend the query to work for multiple servers.

If we use,
index=_internal source=*splunkd.log* | stats count by host .. It may not work as host is down and won't be included in the results set.

Re: Alert when a Splunk service is down

pradeepkumarg — Fri, 27 Apr 2018 02:18:00 GMT

You can search metadata and alert if forwarders do not report for more than a certain threshold

| metadata type=hosts | eval age = now() - lastTime | search age > 300

Re: Alert when a Splunk service is down

Yorokobi — Fri, 27 Apr 2018 03:22:47 GMT

Add your heavy forwarders as search peers to your Monitoring Console and enable the "DMC Alert - Search Peer Not Responding" alert.

Re: Alert when a Splunk service is down

nawazns5038 — Fri, 27 Apr 2018 23:09:18 GMT

However, in environments with large numbers of values for each category, the data might not be complete. This is intentional and allows the metadata command to operate within reasonable time and memory usage. ... from docs.

I don't think metadata can produce accurate results, I don't see it working

Re: Alert when a Splunk service is down

nawazns5038 — Fri, 27 Apr 2018 23:12:00 GMT

It checks only the indexers and that too only the management port (8089)

Re: Alert when a Splunk service is down

xpac — Mon, 30 Apr 2018 21:15:26 GMT

@Yorokobi is right - if you add the HFs as search peers on your Monitoring console, the MC will contact them via port 8089 and you can use it's built-in alert to get a notification when one of them goes down. Actually works for all Splunk instances, be they indexers, search heads, HFs...

Re: Alert when a Splunk service is down

woodcock — Thu, 10 May 2018 00:03:20 GMT

If you use what you have and add one more line, you have an instant alert.

index=_internal source=*splunkd.log*
| stats dc(host) AS count values(host)
| where count < <known_number_of_hosts>

Re: Alert when a Splunk service is down

somesoni2 — Thu, 10 May 2018 00:35:11 GMT

This needs one more stats or (just) dc(host) on existing one. Right now the count gives count of evwnts for host.

Re: Alert when a Splunk service is down

woodcock — Thu, 10 May 2018 14:38:53 GMT

Correct, I really messed that up the first time. Corrected now.

Re: Alert when a Splunk service is down

nawazns5038 — Tue, 29 Sep 2020 21:04:50 GMT

Create a lookup with all the required hostnames and use it in the below query.

Re: Alert when a Splunk service is down

Krishnagrandhi — Thu, 21 Feb 2019 16:30:20 GMT

Also you can narrow down search | metadata type=hosts | search host= | eval age = now() - lastTime | search age > 300

| metadata type=hosts | search host=testweb* | eval age = now() - lastTime | search age > 300