Join the Conversation
- Find Answers
- :
- Splunk Administration
- :
- Getting Data In
- :
- Re: Missing forwarders showing incorrect results.
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Missing forwarders showing incorrect results.
Hi,
Within DMC there is Missing forwarders alert and the alert is flagging one of the host as missing but we can see data coming from that host in splunk. Its incorrect result from the alert, can someone provide an insight as to why this would be happening.
| inputlookup dmc_forwarder_assets
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
i think that this lookup is being updated every period of time (interval), maybe the forwarder was down, the DMC picked it as "down" and it came back up before the now search that populates the lookup was fired ...
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Have you tried to add it to a custom group then removing the custome group/label?
Not sure, why but it seems an ongoing bug
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
hello @Juhi28 and @valiquet,
I was facing the same problem as you. I was getting several Forwared "missing" alerts because I had done the Forwared reboot on the server and Splunk assigned a new GUID for the installation making the server think the old one was inaccessible. To resolve this issue you need to do the following: Settings> Monitoring Console> Settings> Forwarder Monitoring Setup and click on "Rebuild Forwarder assets" you will see that the alerts are gone.
you can also check that this Forwared are saved in the following file: /opt/splunk/etc/apps/splunk_monitoring_console/lookups/dmc_forwarder_assets.csv after Rebuild you will see that Splunk has removed the assets that were missing status
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
"Rebuild Forwarder assets" was the simple manual fix that worked this time around, forgot about that configuration option. Will have to monitor to see if this reappears as I would not want to rely on this as a step to validate the missing forwarders alert. Thanks.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is resolved.
rebuilding asset table for last 4hours [or less] data updates the status of fwders.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
yes forwarder was configured to collect data every 24 hours so was showing incorrect results even when it was up. Also curious to know if we can configure forwarders to collect data hourly [instead of 24 hours] so that DMC gives us an updated stats. ie. Data Collection Interval = hourly
Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!
Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.
Agent Mode Engaged! Enchaining Agentic Operations with Splunk AI Assistant 2.0
Announcing Modern Navigation: A New Era of Splunk User Experience
Modernize your Splunk Apps – Introducing Python 3.13 in Splunk
