DM missing forwarders (ALL)

[splunkreal]

Hello guys,

since 6.5.2 I regularly receive this kind of (fake) alert :

The alert condition for 'DM missing forwarders' was triggered.
Forwarder,Source IP,Forwarder Type,Last Connected,Current Status
(ALL),(ALL),Splunk forwarder,1493074800,missing

Thanks for your help!

[dineshraj9]

You can disable the alert "DMC missing forwarders" under "splunk_monitoring_console" app.

However, it is a useful search which lists the forwarders which are not reporting to your environment any longer. You can optimize the search to show only those servers which have not responded for more than an hour, in last 24 hours by modifying the search to -

| inputlookup dmc_forwarder_assets
| search status="missing" 
| rename hostname as Instance | eval diff=round(now()-last_connected) | where diff>3600 AND diff<86400 | convert ctime(last_connected) 

This way the list will be small and you can restart the forwarder instances which are down.

[splunkreal]

does it mean this is a legitimate alert? all forwarders are disconnected at once?

[dineshraj9]

Is your DMC connected to all indexers that are receiving data from the forwarders?
In our environment we have our custom search to monitor such forwarder issues.

| metadata type=hosts | eval diff=now()-lastTime | where diff > 3600 AND diff < 86400 | convert ctime(lastTime) | table host lastTime

Check if the results match.

[gcusello]

Hi realsplunk,
if you go in http://your_server:8000/en-US/app/splunk_monitoring_console/alerts
you can find all the alerts of Distributed Monitoring Console.
Go in "DMC Alert - Missing forwarders" Alert and disable it.
Bye.
Giuseppe