topic Re: DM missing forwarders (ALL) in Alerting

DM missing forwarders (ALL)

splunkreal — Tue, 25 Apr 2017 10:12:12 GMT

Hello guys,

since 6.5.2 I regularly receive this kind of (fake) alert :

The alert condition for 'DM missing forwarders' was triggered.
Forwarder,Source IP,Forwarder Type,Last Connected,Current Status
(ALL),(ALL),Splunk forwarder,1493074800,missing

Thanks for your help!

Re: DM missing forwarders (ALL)

gcusello — Tue, 25 Apr 2017 10:19:20 GMT

Hi realsplunk,
if you go in http://your_server:8000/en-US/app/splunk_monitoring_console/alerts
you can find all the alerts of Distributed Monitoring Console.
Go in "DMC Alert - Missing forwarders" Alert and disable it.
Bye.
Giuseppe

Re: DM missing forwarders (ALL)

dineshraj9 — Tue, 29 Sep 2020 13:49:41 GMT

You can disable the alert "DMC missing forwarders" under "splunk_monitoring_console" app.

However, it is a useful search which lists the forwarders which are not reporting to your environment any longer. You can optimize the search to show only those servers which have not responded for more than an hour, in last 24 hours by modifying the search to -

| inputlookup dmc_forwarder_assets
| search status="missing" 
| rename hostname as Instance | eval diff=round(now()-last_connected) | where diff>3600 AND diff<86400 | convert ctime(last_connected)

This way the list will be small and you can restart the forwarder instances which are down.

Re: DM missing forwarders (ALL)

splunkreal — Tue, 25 Apr 2017 10:29:16 GMT

does it mean this is a legitimate alert? all forwarders are disconnected at once?

Re: DM missing forwarders (ALL)

dineshraj9 — Tue, 25 Apr 2017 10:34:28 GMT

Is your DMC connected to all indexers that are receiving data from the forwarders?
In our environment we have our custom search to monitor such forwarder issues.

| metadata type=hosts | eval diff=now()-lastTime | where diff > 3600 AND diff < 86400 | convert ctime(lastTime) | table host lastTime

Check if the results match.