Within DMC there is Missing forwarders alert and the alert is flagging one of the host as missing but we can see data coming from that host in splunk. Its incorrect result from the alert, can someone provide an insight as to why this would be happening.
| inputlookup dmc_forwarder_assets
i think that this lookup is being updated every period of time (interval), maybe the forwarder was down, the DMC picked it as "down" and it came back up before the now search that populates the lookup was fired ...
hello @Juhi28 and @valiquet,
I was facing the same problem as you. I was getting several Forwared "missing" alerts because I had done the Forwared reboot on the server and Splunk assigned a new GUID for the installation making the server think the old one was inaccessible. To resolve this issue you need to do the following: Settings> Monitoring Console> Settings> Forwarder Monitoring Setup and click on "Rebuild Forwarder assets" you will see that the alerts are gone.
you can also check that this Forwared are saved in the following file: /opt/splunk/etc/apps/splunk_monitoring_console/lookups/dmc_forwarder_assets.csv after Rebuild you will see that Splunk has removed the assets that were missing status
"Rebuild Forwarder assets" was the simple manual fix that worked this time around, forgot about that configuration option. Will have to monitor to see if this reappears as I would not want to rely on this as a step to validate the missing forwarders alert. Thanks.
yes forwarder was configured to collect data every 24 hours so was showing incorrect results even when it was up. Also curious to know if we can configure forwarders to collect data hourly [instead of 24 hours] so that DMC gives us an updated stats. ie. Data Collection Interval = hourly