Why is the Distributed Management Console unable to find forwarders?

[tom8h]

I configured Forwarder Monitoring Setup of DMC function for monitoring status of forwarders, but the Distributed Management Console (DMC) can't find the forwarders at "Forwarders: Instance" with "Search is waiting for input" message.

I didn't register indexers to the distributed search peers of DMC because the indexers are clustered.
I didn't embed DMC function in cluster master.
By the way, the indexers can receive date accurately from forwarders and cluster master without DMC function can manage the indexers (replication, indexer discovery, and so on).

I'm guessing that the DMC can't search information of clustered indexers from cluster master because the indexers did not forward the information about forwarders or the search statement of DMC (Forwarders: Instance) didn't work.

Could you kindly tell me how to resolve the above?

[lguinn2 · Accepted solution]

The DMC should be a search head on your indexer cluster. If you set that up, the DMC should be able to find everything it needs to know about the forwarders, because they forward their internal logs to the indexers.

You are correct - you should not use distributed search from the DMC to the indexers.

BTW, to give the DMC the maximum information (and to make your environment easier to debug in general), you should be forwarding the internal logs from your search head, the cluster master, the license master, the deployment server, etc. etc. to the indexer tier. The example outputs.conf at Best practice: Forward search head data to the indexer layer should work fine - just insert your list of indexers...
(I give you a tip of my hat if you already doing this - I am on a bit of a mission to get more people to do it)

[tom8h]

Thank you for your answer.

I forgot to describe one thing:
My DMC is embedded with Deployer in my environment. In my understanding, the Deployer should be out of search head cluster, and search head cluster nodes should be on the indexer cluster for distributed search.

Can the Deployer (with DMC) be a search head on the indexer cluster without belonging to search head cluster?

[lguinn2]

Yes, the Deployer + DMC can be a search head on your indexer cluster. In fact, I think it should be.
And you are right, neither the Deployer nor the DMC can be a member of the search head cluster.

But you can search an indexer cluster with a mix of independent search heads (like the DMC or Deployer) and search heads in a Search Head Cluster. It all works.

FInally, I would probably word this differently "search head cluster nodes should be on the indexer cluster for distributed search." It is a mouthful. But what I really want to say is: Any search head - clustered or not - can become a participant in an indexer cluster. The search head just needs to register itself with the cluster master and provide the secret password. The search head does not need to configure distributed search to search an indexer cluster.

HTH