Splunk 6.2 Deployment Monitor repeatedly sends "forwarder missing" alert emails

[droth333]

Immediately after upgrading from 6.0 to 6.2 Indexer, we get "missing forwarder" alerts from Deployment Monitor with
subject: [SPLUNK]: DM missing forwarders.

These repeat every two hours and include every existing forwarder (which are confirmed to all be running, tailing logs, sending log data, and indexing logged data on the Indexer). One clue is that instead of listing the symbolic hostname in the "Forwarder" column (as it always has in the past), it lists the IP address.

In other words, it appears that all the existing forwarders got "duplicated" in metrics logs with their IP addresses instead of their
symbolic hostnames (like webserver.mycompany.com).
And that the Deployment Monitor thinks these are now all "missing" (maybe because all forwarders send with host=symbolic_name).

I am NOT running the Deployment Mgr itself.

Thanks!

[ibondarets]

Hi!
How could I set up this email alerts on missing forwarders? I'd like to receive both realtime alerts and a daily report on missing agents.
I tried to use search from Distributed Management Console:

| inputlookup dmc_forwarder_assets | makemv delim=" " avg_tcp_kbps_sparkline | eval sum_kb = if (status == "missing", "N/A", sum_kb) | eval avg_tcp_kbps_sparkline = if (status == "missing", "N/A", avg_tcp_kbps_sparkline) | eval avg_tcp_kbps = if (status == "missing", "N/A", avg_tcp_kbps) | eval avg_tcp_eps = if (status == "missing", "N/A", avg_tcp_eps) | dmc_rename_forwarder_type(forwarder_type) | dmc_time_format(last_connected) | fields hostname, forwarder_type, version, os, arch, status, last_connected, sum_kb, avg_tcp_kbps_sparkline, avg_tcp_kbps, avg_tcp_eps | search hostname="***" | search status="missing" | rename hostname as Instance, forwarder_type as Type, version as Version, os as OS, arch as Architecture, status as Status, last_connected as "Last Connected to Indexers", sum_kb as "Total KB", avg_tcp_kbps_sparkline as "Average KB/s Over Time", avg_tcp_kbps as "Average KB/s", avg_tcp_eps as "Average Events/s"
but it only works when run within DMC, if i try to create a report out of it - it doesn't work, I guess it's because lookup table is under DMC app:
/opt/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv

How can I build a scheduled report and a realtime alert for my goal?

[JohnBACSplunk]

Please see http://answers.splunk.com/answers/188784/after-update-to-splunk-enterprise-62-why-does-the.html for the answer.
To Summarize: It is a product defect, I believe for the deployment monitor. Cause: In Splunk Enterprise 6.2, indexers are logging new events to metrics.log/group=tcpin_connections to record forwarder connection events, such as a connection closing.

Fix is to change macros.conf in deployment monitor. Details are here