Getting Data In

Missing forwarders showing incorrect results.

Juhi28
New Member

Hi,

Within DMC there is Missing forwarders alert and the alert is flagging one of the host as missing but we can see data coming from that host in splunk. Its incorrect result from the alert, can someone provide an insight as to why this would be happening.

| inputlookup dmc_forwarder_assets

Tags (1)
0 Karma

adonio
Ultra Champion

i think that this lookup is being updated every period of time (interval), maybe the forwarder was down, the DMC picked it as "down" and it came back up before the now search that populates the lookup was fired ...

0 Karma

valiquet
Contributor

Have you tried to add it to a custom group then removing the custome group/label?

Not sure, why but it seems an ongoing bug

0 Karma

justodaniel
Path Finder

hello @Juhi28 and @valiquet,
I was facing the same problem as you. I was getting several Forwared "missing" alerts because I had done the Forwared reboot on the server and Splunk assigned a new GUID for the installation making the server think the old one was inaccessible. To resolve this issue you need to do the following: Settings> Monitoring Console> Settings> Forwarder Monitoring Setup and click on "Rebuild Forwarder assets" you will see that the alerts are gone.

you can also check that this Forwared are saved in the following file: /opt/splunk/etc/apps/splunk_monitoring_console/lookups/dmc_forwarder_assets.csv after Rebuild you will see that Splunk has removed the assets that were missing status

kevincmartin
Engager

"Rebuild Forwarder assets" was the simple manual fix that worked this time around, forgot about that configuration option. Will have to monitor to see if this reappears as I would not want to rely on this as a step to validate the missing forwarders alert. Thanks.

0 Karma

Juhi28
New Member

This is resolved.

rebuilding asset table for last 4hours [or less] data updates the status of fwders.

0 Karma

Juhi28
New Member

yes forwarder was configured to collect data every 24 hours so was showing incorrect results even when it was up. Also curious to know if we can configure forwarders to collect data hourly [instead of 24 hours] so that DMC gives us an updated stats. ie. Data Collection Interval = hourly

0 Karma
Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...