Monitoring Splunk

Splunk 6.2 Deployment Monitor repeatedly sends "forwarder missing" alert emails


Immediately after upgrading from 6.0 to 6.2 Indexer, we get "missing forwarder" alerts from Deployment Monitor with
subject: [SPLUNK]: DM missing forwarders.

These repeat every two hours and include every existing forwarder (which are confirmed to all be running, tailing logs, sending log data, and indexing logged data on the Indexer). One clue is that instead of listing the symbolic hostname in the "Forwarder" column (as it always has in the past), it lists the IP address.

In other words, it appears that all the existing forwarders got "duplicated" in metrics logs with their IP addresses instead of their
symbolic hostnames (like
And that the Deployment Monitor thinks these are now all "missing" (maybe because all forwarders send with host=symbolic_name).

I am NOT running the Deployment Mgr itself.



How could I set up this email alerts on missing forwarders? I'd like to receive both realtime alerts and a daily report on missing agents.
I tried to use search from Distributed Management Console:

| inputlookup dmc_forwarder_assets | makemv delim=" " avg_tcp_kbps_sparkline | eval sum_kb = if (status == "missing", "N/A", sum_kb) | eval avg_tcp_kbps_sparkline = if (status == "missing", "N/A", avg_tcp_kbps_sparkline) | eval avg_tcp_kbps = if (status == "missing", "N/A", avg_tcp_kbps) | eval avg_tcp_eps = if (status == "missing", "N/A", avg_tcp_eps) | dmc_rename_forwarder_type(forwarder_type) | dmc_time_format(last_connected) | fields hostname, forwarder_type, version, os, arch, status, last_connected, sum_kb, avg_tcp_kbps_sparkline, avg_tcp_kbps, avg_tcp_eps | search hostname="***" | search status="missing" | rename hostname as Instance, forwarder_type as Type, version as Version, os as OS, arch as Architecture, status as Status, last_connected as "Last Connected to Indexers", sum_kb as "Total KB", avg_tcp_kbps_sparkline as "Average KB/s Over Time", avg_tcp_kbps as "Average KB/s", avg_tcp_eps as "Average Events/s"
but it only works when run within DMC, if i try to create a report out of it - it doesn't work, I guess it's because lookup table is under DMC app:

How can I build a scheduled report and a realtime alert for my goal?

0 Karma


Please see for the answer.
To Summarize: It is a product defect, I believe for the deployment monitor. Cause: In Splunk Enterprise 6.2, indexers are logging new events to metrics.log/group=tcpin_connections to record forwarder connection events, such as a connection closing.

Fix is to change macros.conf in deployment monitor. Details are here

0 Karma
Get Updates on the Splunk Community!

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...