Deployment Architecture

How do I remove "missing" forwarders from Splunk Deployment Monitor 4.3.1?

Cagey
Engager

Every time I go into deployment monitor it tells me I have 65 missing forwarders. In all cases these forwarders are listed as an IP address. In some cases the IP address corresponds to a "active" forwarder which is reported by the servers name. In other cases the forwarder is actually no longer in service and needs to be removed from the list of forwarders. I have read other comments regarding this and they mention a forwarder as going "quiet" or deployment monitor have a "remove missing forwarders" button. In my case neither of these is present.

As I see it this is actually two problems:
1. making splunk correlate the IP address of the "missing" forwarder to the DNS name for the associated "active" forwarder.
2. remove actual "missing" forwarders from the list of forwarders.

gpullis
Communicator

What worked for me was using the Rebuild forwarder assets... button in Monitoring Console > Settings > Forwarder Monitoring Setup.

See: https://docs.splunk.com/Documentation/Splunk/7.1.1/DMC/Configureforwardermonitoring

richaGindodia
Path Finder

Not sure of this. But you could actually add a ping script to your forwarders which would ping your server at regular intervals.

0 Karma

Cagey
Engager

Thank you for your response Rich but this would not solve my problem. All the forwarders report to an indexing server which keeps track, via a database or something, of all the forwarders and when they last reported into the indexer. Now my problem (which actually has two parts) is that I cannot acknowledge the missing forwarders so that they stop showing up in the list of forwarders.

To further explain the first part of my problem, suppose I have a forwarder with a DNS name of "forwarder1" and an IP address of "1.2.3.4". My indexer is reporting that "forwarder1" is active but IP address "1.2.3.4" is missing. This is not possible since they are the same device. Obviously this is a problem with the actual code or database which is used to report the forwarders.

The second part of the problem is that I DO actually have some forwarders which are no longer in service and they are rightly being reported as missing. However, I know this and would like to acknowledge this to the application and stop having them reported as missing. The problem is, there is no way to do this so every time I go into the application I am once again informed about the missing forwarders. However, if there are any new ones listed it is hard to pick them out from the large list of 65.

So, still two problems:

  1. Code (or database) needs fixing to correlate the IP with the DNS name.
  2. Acknowledgement function required to remove actual "missing" forwarders from the database.
0 Karma

gpullis
Communicator

Yeah. Same. We're logging VDI machines that are pretty ephemeral, so my production indexer is complaining about 4758 "missing" forwarders. Some of those are legit, but it's pretty painful to try to figure out which ones.

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...