Monitoring Splunk

Splunk 6.2 Deployment Monitor repeatedly sends "forwarder missing" alert emails

droth333
Explorer

Immediately after upgrading from 6.0 to 6.2 Indexer, we get "missing forwarder" alerts from Deployment Monitor with
subject: [SPLUNK]: DM missing forwarders.

These repeat every two hours and include every existing forwarder (which are confirmed to all be running, tailing logs, sending log data, and indexing logged data on the Indexer). One clue is that instead of listing the symbolic hostname in the "Forwarder" column (as it always has in the past), it lists the IP address.

In other words, it appears that all the existing forwarders got "duplicated" in metrics logs with their IP addresses instead of their
symbolic hostnames (like webserver.mycompany.com).
And that the Deployment Monitor thinks these are now all "missing" (maybe because all forwarders send with host=symbolic_name).

I am NOT running the Deployment Mgr itself.

Thanks!

ibondarets
Explorer

Hi!
How could I set up this email alerts on missing forwarders? I'd like to receive both realtime alerts and a daily report on missing agents.
I tried to use search from Distributed Management Console:

| inputlookup dmc_forwarder_assets | makemv delim=" " avg_tcp_kbps_sparkline | eval sum_kb = if (status == "missing", "N/A", sum_kb) | eval avg_tcp_kbps_sparkline = if (status == "missing", "N/A", avg_tcp_kbps_sparkline) | eval avg_tcp_kbps = if (status == "missing", "N/A", avg_tcp_kbps) | eval avg_tcp_eps = if (status == "missing", "N/A", avg_tcp_eps) | dmc_rename_forwarder_type(forwarder_type) | dmc_time_format(last_connected) | fields hostname, forwarder_type, version, os, arch, status, last_connected, sum_kb, avg_tcp_kbps_sparkline, avg_tcp_kbps, avg_tcp_eps | search hostname="***" | search status="missing" | rename hostname as Instance, forwarder_type as Type, version as Version, os as OS, arch as Architecture, status as Status, last_connected as "Last Connected to Indexers", sum_kb as "Total KB", avg_tcp_kbps_sparkline as "Average KB/s Over Time", avg_tcp_kbps as "Average KB/s", avg_tcp_eps as "Average Events/s"
but it only works when run within DMC, if i try to create a report out of it - it doesn't work, I guess it's because lookup table is under DMC app:
/opt/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv

How can I build a scheduled report and a realtime alert for my goal?

0 Karma

JohnBACSplunk
Engager

Please see http://answers.splunk.com/answers/188784/after-update-to-splunk-enterprise-62-why-does-the.html for the answer.
To Summarize: It is a product defect, I believe for the deployment monitor. Cause: In Splunk Enterprise 6.2, indexers are logging new events to metrics.log/group=tcpin_connections to record forwarder connection events, such as a connection closing.

Fix is to change macros.conf in deployment monitor. Details are here

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...