Every time I go into deployment monitor it tells me I have 65 missing forwarders. In all cases these forwarders are listed as an IP address. In some cases the IP address corresponds to a "active" forwarder which is reported by the servers name. In other cases the forwarder is actually no longer in service and needs to be removed from the list of forwarders. I have read other comments regarding this and they mention a forwarder as going "quiet" or deployment monitor have a "remove missing forwarders" button. In my case neither of these is present.
As I see it this is actually two problems:
1. making splunk correlate the IP address of the "missing" forwarder to the DNS name for the associated "active" forwarder.
2. remove actual "missing" forwarders from the list of forwarders.
What worked for me was using the Rebuild forwarder assets... button in Monitoring Console > Settings > Forwarder Monitoring Setup.
See: https://docs.splunk.com/Documentation/Splunk/7.1.1/DMC/Configureforwardermonitoring
Not sure of this. But you could actually add a ping script to your forwarders which would ping your server at regular intervals.
Thank you for your response Rich but this would not solve my problem. All the forwarders report to an indexing server which keeps track, via a database or something, of all the forwarders and when they last reported into the indexer. Now my problem (which actually has two parts) is that I cannot acknowledge the missing forwarders so that they stop showing up in the list of forwarders.
To further explain the first part of my problem, suppose I have a forwarder with a DNS name of "forwarder1" and an IP address of "1.2.3.4". My indexer is reporting that "forwarder1" is active but IP address "1.2.3.4" is missing. This is not possible since they are the same device. Obviously this is a problem with the actual code or database which is used to report the forwarders.
The second part of the problem is that I DO actually have some forwarders which are no longer in service and they are rightly being reported as missing. However, I know this and would like to acknowledge this to the application and stop having them reported as missing. The problem is, there is no way to do this so every time I go into the application I am once again informed about the missing forwarders. However, if there are any new ones listed it is hard to pick them out from the large list of 65.
So, still two problems:
Yeah. Same. We're logging VDI machines that are pretty ephemeral, so my production indexer is complaining about 4758 "missing" forwarders. Some of those are legit, but it's pretty painful to try to figure out which ones.