Getting Data In

Why is the Distributed Management Console unable to find forwarders?

tom8h
Explorer

I configured Forwarder Monitoring Setup of DMC function for monitoring status of forwarders, but the Distributed Management Console (DMC) can't find the forwarders at "Forwarders: Instance" with "Search is waiting for input" message.

I didn't register indexers to the distributed search peers of DMC because the indexers are clustered.
I didn't embed DMC function in cluster master.
By the way, the indexers can receive date accurately from forwarders and cluster master without DMC function can manage the indexers (replication, indexer discovery, and so on).

I'm guessing that the DMC can't search information of clustered indexers from cluster master because the indexers did not forward the information about forwarders or the search statement of DMC (Forwarders: Instance) didn't work.

Could you kindly tell me how to resolve the above?

0 Karma
1 Solution

lguinn2
Legend

The DMC should be a search head on your indexer cluster. If you set that up, the DMC should be able to find everything it needs to know about the forwarders, because they forward their internal logs to the indexers.

You are correct - you should not use distributed search from the DMC to the indexers.

BTW, to give the DMC the maximum information (and to make your environment easier to debug in general), you should be forwarding the internal logs from your search head, the cluster master, the license master, the deployment server, etc. etc. to the indexer tier. The example outputs.conf at Best practice: Forward search head data to the indexer layer should work fine - just insert your list of indexers...
(I give you a tip of my hat if you already doing this - I am on a bit of a mission to get more people to do it)

View solution in original post

lguinn2
Legend

The DMC should be a search head on your indexer cluster. If you set that up, the DMC should be able to find everything it needs to know about the forwarders, because they forward their internal logs to the indexers.

You are correct - you should not use distributed search from the DMC to the indexers.

BTW, to give the DMC the maximum information (and to make your environment easier to debug in general), you should be forwarding the internal logs from your search head, the cluster master, the license master, the deployment server, etc. etc. to the indexer tier. The example outputs.conf at Best practice: Forward search head data to the indexer layer should work fine - just insert your list of indexers...
(I give you a tip of my hat if you already doing this - I am on a bit of a mission to get more people to do it)

tom8h
Explorer

Thank you for your answer.

I forgot to describe one thing:
My DMC is embedded with Deployer in my environment. In my understanding, the Deployer should be out of search head cluster, and search head cluster nodes should be on the indexer cluster for distributed search.

Can the Deployer (with DMC) be a search head on the indexer cluster without belonging to search head cluster?

0 Karma

lguinn2
Legend

Yes, the Deployer + DMC can be a search head on your indexer cluster. In fact, I think it should be.
And you are right, neither the Deployer nor the DMC can be a member of the search head cluster.

But you can search an indexer cluster with a mix of independent search heads (like the DMC or Deployer) and search heads in a Search Head Cluster. It all works.

FInally, I would probably word this differently "search head cluster nodes should be on the indexer cluster for distributed search." It is a mouthful. But what I really want to say is: Any search head - clustered or not - can become a participant in an indexer cluster. The search head just needs to register itself with the cluster master and provide the secret password. The search head does not need to configure distributed search to search an indexer cluster.

HTH

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...