topic When accessing the Distributed Management Console, why is there no data for any search heads in our distributed search environment? in Deployment Architecture

When accessing the Distributed Management Console, why is there no data for any search heads in our distributed search environment?

tlmayes — Tue, 29 Sep 2020 09:24:24 GMT

We are attempting to deploy the Distributed Management Console, but are not having success. I have reviewed all of the previous submissions, but am not finding a solution to our challenge. We are running Splunk 6.3.2 in a distributed environment. I have followed the DMC deployment doc for the DMC prerequisites, but have obviously missed something. All nodes have been added in the DMC as Search Peers, _introspection_generator configurations have been verified, platform instrumentation has been enabled.

The problem: When accessing the DMC console, there is no data for any of the Search Heads in any of the DMC screens. All other systems appear, and with report content.

I have an environment consisting of 14 nodes:
4 indexers
4 forwarders
3 search heads
3 management servers: Deployment Svr, Search Head Cluster Deployer, Indexer Cluster Master/DMC

Thanks in advance

Re: When accessing the Distributed Management Console, why is there no data for any search heads in our distributed search environment?

ykou_splunk — Wed, 13 Apr 2016 16:59:35 GMT

There are two possible reasons:

you didn't switch DMC to distributed mode in DMC General Setup page.
you didn't forward all search heads' internal logs to indexers.

The first reason is more likely to be your case, otherwise the DMC dashboards' Snapshot section should work because it doesn't rely on internal logs. In addition, please verify that the search head names showing up in the Instance dropdown menu on the top of each dashboard.

Re: When accessing the Distributed Management Console, why is there no data for any search heads in our distributed search environment?

tlmayes — Wed, 13 Apr 2016 17:29:51 GMT

Appreciate the quick turnaround on a response. I failed to add in my post that I: Switched from "standalone" to "distributed" on the DMC in "Settings > Distributed Management Console > Settings > General Setup". So this is done and confirmed, again 🙂

On the second, have always thought this might be the case, but not quite sure how to verify other than performing a search in index=*_ and filtering for the Search Heads (which by the way returns nothing). Configured "outputs.conf" on the Search Heads per the DMC docs with the server line pointing to my Indexers.

Regarding the dashboards and the "Snapshot", I assume you are referring to the "Overview". Nothing there, or within any of the dropdowns under "Search", or anywhere else that is reporting on "Search" data.

Regarding the Instances dropdown in DMC, the search heads do NOT show up here.

Re: When accessing the Distributed Management Console, why is there no data for any search heads in our distributed search environment?

ykou_splunk — Wed, 13 Apr 2016 18:14:55 GMT

when you switched to "distributed" mode, do you see a table of instances listed on that page? your search heads should appear in that table. Also, make sure they have the "Search Head" role. Then click the "Apply changes" button on top right corner on that page.

Please note assigning the right role and clicking the "Apply changes" button are required steps. Once these steps are done, at least the search heads should show up in the Instances dropdown.

Re: When accessing the Distributed Management Console, why is there no data for any search heads in our distributed search environment?

tlmayes — Wed, 13 Apr 2016 18:22:57 GMT

Yes, when I switched to distributed mode the Search Heads did (and do) show up listed as instances, and the column Search Head Cluster(s) was populated as well.

The following details appear
Server Role: KV Store / Search Head
Search Head Cluster(s):
Monitoring: Enabled
State: Configured

Re: When accessing the Distributed Management Console, why is there no data for any search heads in our distributed search environment?

tlmayes — Wed, 13 Apr 2016 20:24:05 GMT

Clarification to previous post: The DMC was already in Distributed mode, I did not change it into Distributed mode, and although the Search Heads appear as described (already did) they still do not appear in the Distributed Management Console in any of the reporting.

Re: When accessing the Distributed Management Console, why is there no data for any search heads in our distributed search environment?

ykou_splunk — Wed, 13 Apr 2016 21:01:31 GMT

when you click the "apply changes" button, is there a dialog popup showing the saving progress and finally shows a confirmation that the changes have been saved?

Re: When accessing the Distributed Management Console, why is there no data for any search heads in our distributed search environment?

tlmayes — Thu, 14 Apr 2016 12:12:29 GMT

Seems I do not have enough "carma" points to post an attachment 😞

The dialog does appear, and completes the process. Each time I have tried this, I receive slightly different errors.

This time:

"At least one of your instances is a deployment server plus other non-deployer roles. We recommend only deployer roles per deployment server"

"At least one of your instances is a search head deployer plus other non-deployer roles. We recommend only deployer roles per search head deployer"

"At least one of your instances is an indexer plus other roles. We recommend only one role per indexer"

"At lease one of your instances is a search head deployer without a search head cluster label. We recommendyou edit these instances to set their search head cluster labels"

Roles as displayed in the DMC "Setup Screen"
This Instance (the DMC node)
- Instance: Cluster Master / License Master / Search Head

Remote instances
- Instance 1: Indexer (indexer cluster)
- Instance 2: Indexer (indexer cluster)
- Instance 3: Indexer (indexer cluster)
- Instance 4: Indexer (indexer cluster)
- Instance 5: Deployment Svr / Indexer / SHC Deployer
- Instance 6: Deployment Svr / Indexer
- Instance 7: KV Store / Search Head (search head cluster) (index cluster)
- Instance 8: KV Store / Search Head (search head cluster) (index cluster)
- Instance 9: KV Store / Search Head (search head cluster) (index cluster)

Re: When accessing the Distributed Management Console, why is there no data for any search heads in our distributed search environment?

bandit — Thu, 15 Dec 2016 04:34:23 GMT

I've found on my dedicated DMC instance if I add search head as a search peer, it shows up immediately and classifies it appropriately as a search head. A better setup would be to prompt users for endpoints to monitor.

Rob