Deployment Architecture

How do I remove "missing" forwarders from Splunk Deployment Monitor 4.3.1?

Cagey
Engager

Every time I go into deployment monitor it tells me I have 65 missing forwarders. In all cases these forwarders are listed as an IP address. In some cases the IP address corresponds to a "active" forwarder which is reported by the servers name. In other cases the forwarder is actually no longer in service and needs to be removed from the list of forwarders. I have read other comments regarding this and they mention a forwarder as going "quiet" or deployment monitor have a "remove missing forwarders" button. In my case neither of these is present.

As I see it this is actually two problems:
1. making splunk correlate the IP address of the "missing" forwarder to the DNS name for the associated "active" forwarder.
2. remove actual "missing" forwarders from the list of forwarders.

gpullis
Communicator

What worked for me was using the Rebuild forwarder assets... button in Monitoring Console > Settings > Forwarder Monitoring Setup.

See: https://docs.splunk.com/Documentation/Splunk/7.1.1/DMC/Configureforwardermonitoring

richaGindodia
Path Finder

Not sure of this. But you could actually add a ping script to your forwarders which would ping your server at regular intervals.

0 Karma

Cagey
Engager

Thank you for your response Rich but this would not solve my problem. All the forwarders report to an indexing server which keeps track, via a database or something, of all the forwarders and when they last reported into the indexer. Now my problem (which actually has two parts) is that I cannot acknowledge the missing forwarders so that they stop showing up in the list of forwarders.

To further explain the first part of my problem, suppose I have a forwarder with a DNS name of "forwarder1" and an IP address of "1.2.3.4". My indexer is reporting that "forwarder1" is active but IP address "1.2.3.4" is missing. This is not possible since they are the same device. Obviously this is a problem with the actual code or database which is used to report the forwarders.

The second part of the problem is that I DO actually have some forwarders which are no longer in service and they are rightly being reported as missing. However, I know this and would like to acknowledge this to the application and stop having them reported as missing. The problem is, there is no way to do this so every time I go into the application I am once again informed about the missing forwarders. However, if there are any new ones listed it is hard to pick them out from the large list of 65.

So, still two problems:

  1. Code (or database) needs fixing to correlate the IP with the DNS name.
  2. Acknowledgement function required to remove actual "missing" forwarders from the database.
0 Karma

gpullis
Communicator

Yeah. Same. We're logging VDI machines that are pretty ephemeral, so my production indexer is complaining about 4758 "missing" forwarders. Some of those are legit, but it's pretty painful to try to figure out which ones.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...

Design, Compete, Win: Submit Your Best Splunk Dashboards for a .conf26 Pass

Hello Splunkers,  We’re excited to kick off a Splunk Dashboard contest! We know that dashboards are a primary ...

May 2026 Splunk Expert Sessions: Security & Observability

Level Up Your Operations: May 2026 Splunk Expert Sessions Whether you are refining your security posture or ...