Getting Data In

Missing forwarders showing incorrect results.

Juhi28
New Member

Hi,

Within DMC there is Missing forwarders alert and the alert is flagging one of the host as missing but we can see data coming from that host in splunk. Its incorrect result from the alert, can someone provide an insight as to why this would be happening.

| inputlookup dmc_forwarder_assets

Tags (1)
0 Karma

adonio
SplunkTrust
SplunkTrust

i think that this lookup is being updated every period of time (interval), maybe the forwarder was down, the DMC picked it as "down" and it came back up before the now search that populates the lookup was fired ...

0 Karma

valiquet
Contributor

Have you tried to add it to a custom group then removing the custome group/label?

Not sure, why but it seems an ongoing bug

0 Karma

justodaniel
Path Finder

hello @Juhi28 and @valiquet,
I was facing the same problem as you. I was getting several Forwared "missing" alerts because I had done the Forwared reboot on the server and Splunk assigned a new GUID for the installation making the server think the old one was inaccessible. To resolve this issue you need to do the following: Settings> Monitoring Console> Settings> Forwarder Monitoring Setup and click on "Rebuild Forwarder assets" you will see that the alerts are gone.

you can also check that this Forwared are saved in the following file: /opt/splunk/etc/apps/splunk_monitoring_console/lookups/dmc_forwarder_assets.csv after Rebuild you will see that Splunk has removed the assets that were missing status

kevincmartin
Engager

"Rebuild Forwarder assets" was the simple manual fix that worked this time around, forgot about that configuration option. Will have to monitor to see if this reappears as I would not want to rely on this as a step to validate the missing forwarders alert. Thanks.

0 Karma

Juhi28
New Member

This is resolved.

rebuilding asset table for last 4hours [or less] data updates the status of fwders.

0 Karma

Juhi28
New Member

yes forwarder was configured to collect data every 24 hours so was showing incorrect results even when it was up. Also curious to know if we can configure forwarders to collect data hourly [instead of 24 hours] so that DMC gives us an updated stats. ie. Data Collection Interval = hourly

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!