Deployment Architecture

When accessing the Distributed Management Console, why is there no data for any search heads in our distributed search environment?

tlmayes
Contributor

We are attempting to deploy the Distributed Management Console, but are not having success. I have reviewed all of the previous submissions, but am not finding a solution to our challenge. We are running Splunk 6.3.2 in a distributed environment. I have followed the DMC deployment doc for the DMC prerequisites, but have obviously missed something. All nodes have been added in the DMC as Search Peers, _introspection_generator configurations have been verified, platform instrumentation has been enabled.

The problem: When accessing the DMC console, there is no data for any of the Search Heads in any of the DMC screens. All other systems appear, and with report content.

I have an environment consisting of 14 nodes:
4 indexers
4 forwarders
3 search heads
3 management servers: Deployment Svr, Search Head Cluster Deployer, Indexer Cluster Master/DMC

Thanks in advance

bandit
Motivator

I've found on my dedicated DMC instance if I add search head as a search peer, it shows up immediately and classifies it appropriately as a search head. A better setup would be to prompt users for endpoints to monitor.

Rob

0 Karma

ykou_splunk
Splunk Employee
Splunk Employee

There are two possible reasons:

  • you didn't switch DMC to distributed mode in DMC General Setup page.
  • you didn't forward all search heads' internal logs to indexers.

The first reason is more likely to be your case, otherwise the DMC dashboards' Snapshot section should work because it doesn't rely on internal logs. In addition, please verify that the search head names showing up in the Instance dropdown menu on the top of each dashboard.

tlmayes
Contributor

Yes, when I switched to distributed mode the Search Heads did (and do) show up listed as instances, and the column Search Head Cluster(s) was populated as well.

The following details appear
Server Role: KV Store / Search Head
Search Head Cluster(s):
Monitoring: Enabled
State: Configured

0 Karma

tlmayes
Contributor

Clarification to previous post: The DMC was already in Distributed mode, I did not change it into Distributed mode, and although the Search Heads appear as described (already did) they still do not appear in the Distributed Management Console in any of the reporting.

0 Karma

ykou_splunk
Splunk Employee
Splunk Employee

when you click the "apply changes" button, is there a dialog popup showing the saving progress and finally shows a confirmation that the changes have been saved?

tlmayes
Contributor

Seems I do not have enough "carma" points to post an attachment 😞

The dialog does appear, and completes the process. Each time I have tried this, I receive slightly different errors.

This time:

"At least one of your instances is a deployment server plus other non-deployer roles. We recommend only deployer roles per deployment server"

"At least one of your instances is a search head deployer plus other non-deployer roles. We recommend only deployer roles per search head deployer"

"At least one of your instances is an indexer plus other roles. We recommend only one role per indexer"

"At lease one of your instances is a search head deployer without a search head cluster label. We recommendyou edit these instances to set their search head cluster labels"

Roles as displayed in the DMC "Setup Screen"
This Instance (the DMC node)
- Instance: Cluster Master / License Master / Search Head

Remote instances
- Instance 1: Indexer (indexer cluster)
- Instance 2: Indexer (indexer cluster)
- Instance 3: Indexer (indexer cluster)
- Instance 4: Indexer (indexer cluster)
- Instance 5: Deployment Svr / Indexer / SHC Deployer
- Instance 6: Deployment Svr / Indexer
- Instance 7: KV Store / Search Head (search head cluster) (index cluster)
- Instance 8: KV Store / Search Head (search head cluster) (index cluster)
- Instance 9: KV Store / Search Head (search head cluster) (index cluster)

0 Karma

tlmayes
Contributor

Appreciate the quick turnaround on a response. I failed to add in my post that I: Switched from "standalone" to "distributed" on the DMC in "Settings > Distributed Management Console > Settings > General Setup". So this is done and confirmed, again 🙂

On the second, have always thought this might be the case, but not quite sure how to verify other than performing a search in index=*_ and filtering for the Search Heads (which by the way returns nothing). Configured "outputs.conf" on the Search Heads per the DMC docs with the server line pointing to my Indexers.

Regarding the dashboards and the "Snapshot", I assume you are referring to the "Overview". Nothing there, or within any of the dropdowns under "Search", or anywhere else that is reporting on "Search" data.

Regarding the Instances dropdown in DMC, the search heads do NOT show up here.

0 Karma

ykou_splunk
Splunk Employee
Splunk Employee

when you switched to "distributed" mode, do you see a table of instances listed on that page? your search heads should appear in that table. Also, make sure they have the "Search Head" role. Then click the "Apply changes" button on top right corner on that page.

Please note assigning the right role and clicking the "Apply changes" button are required steps. Once these steps are done, at least the search heads should show up in the Instances dropdown.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...