topic Re: Missing forwarders showing incorrect results. in Getting Data In

Missing forwarders showing incorrect results.

Juhi28 — Tue, 29 Sep 2020 18:13:14 GMT

Hi,

Within DMC there is Missing forwarders alert and the alert is flagging one of the host as missing but we can see data coming from that host in splunk. Its incorrect result from the alert, can someone provide an insight as to why this would be happening.

| inputlookup dmc_forwarder_assets

Re: Missing forwarders showing incorrect results.

adonio — Wed, 28 Feb 2018 13:17:53 GMT

i think that this lookup is being updated every period of time (interval), maybe the forwarder was down, the DMC picked it as "down" and it came back up before the now search that populates the lookup was fired ...

Re: Missing forwarders showing incorrect results.

Juhi28 — Thu, 01 Mar 2018 04:34:14 GMT

yes forwarder was configured to collect data every 24 hours so was showing incorrect results even when it was up. Also curious to know if we can configure forwarders to collect data hourly [instead of 24 hours] so that DMC gives us an updated stats. ie. Data Collection Interval = hourly

Re: Missing forwarders showing incorrect results.

valiquet — Fri, 09 Mar 2018 09:42:29 GMT

Have you tried to add it to a custom group then removing the custome group/label?

Not sure, why but it seems an ongoing bug

Re: Missing forwarders showing incorrect results.

justodaniel — Tue, 29 Sep 2020 20:23:31 GMT

hello @Juhi28 and @valiquet,
I was facing the same problem as you. I was getting several Forwared "missing" alerts because I had done the Forwared reboot on the server and Splunk assigned a new GUID for the installation making the server think the old one was inaccessible. To resolve this issue you need to do the following: Settings> Monitoring Console> Settings> Forwarder Monitoring Setup and click on "Rebuild Forwarder assets" you will see that the alerts are gone.

you can also check that this Forwared are saved in the following file: /opt/splunk/etc/apps/splunk_monitoring_console/lookups/dmc_forwarder_assets.csv after Rebuild you will see that Splunk has removed the assets that were missing status

Re: Missing forwarders showing incorrect results.

kevincmartin — Tue, 27 Nov 2018 14:28:42 GMT

"Rebuild Forwarder assets" was the simple manual fix that worked this time around, forgot about that configuration option. Will have to monitor to see if this reappears as I would not want to rely on this as a step to validate the missing forwarders alert. Thanks.

Re: Missing forwarders showing incorrect results.

Juhi28 — Tue, 15 Jan 2019 03:54:15 GMT

This is resolved.

rebuilding asset table for last 4hours [or less] data updates the status of fwders.