How to integrate search head cluster into indexer cluster?

[transtrophe]

I am trying to integrate a search head cluster into an indexer cluster. Search head cluster has 3 shc members using search factor = 3. Indexer cluster has 5 indexers serviced by the master with replication factor = 5. SHC captain is properly elected. DMC viewed on the Splunk Web master instance shows the index cluster deployment is functioning properly (viewed in Overview panel), but no Search Head panel is visible (although when launching Overview the Search Head cluster panel presents for a quick moment then disappears).

How can I confirm that the search head cluster got integrated to the indexer cluster? Which configuration file(s) get written when executing the edit cluster-config command?

[Steve_G_]

Here's a topic that describes how to integrate a search head cluster with an indexer cluster:

http://docs.splunk.com/Documentation/Splunk/6.2.2/DistSearch/SHCandindexercluster

[transtrophe]

Additionally, here are the configurations of the server.conf files on each of the shc members (showing just one instance with "sanitized values" for hostnames and secrets:

root@xxx:/home/admin# cat /opt/splunk/etc/system/local/server.conf
[sslConfig]
sslKeysfilePassword = xxxxxxxxxxxx

[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[general]
pass4SymmKey = xxxxxxx
serverName = shc_member1_hostname

[replication_port://9997]

[shclustering]
conf_deploy_fetch_url = https://shc_deployer_hostname
disabled = 0
mgmt_uri = https://shc_member1_hostname
pass4SymmKey = xxxxxxxxx
id = 43CD9A09-DAB9-44DF-BF43-55C0A72324FD

[clustering]
master_uri = https://master_indexer_hostname
mode = searchhead
pass4SymmKey = xxxxxxxxxx

[esix_splunk · Accepted solution]

SHC and Index Clustering are two different components in the Splunk world that do not integrate in relation to sharing functionality.

In order to "integrate" your SHC, I am guessing you mean to include Search Functionality, and perhap Search site affinity?

In the case of Search Functionality, you have to add the search head to the cluster as search peers. Additionally, you can add site member should (via server.conf,) and this will designate the search affinity for the SHC member.

Short of it, add the SHC members to the cluster as search peers. Once done, you should be able to search across the cluster and additionally see the SHC members on the CLuster Master, under search peers.

Additional configuration relating to search affinity can be found here: http://docs.splunk.com/Documentation/Splunk/6.2.2/Indexer/Multisitesearchaffinity

Regarding DMC, this cannot reside on the SHC or its members. You need to create this role outside of the Search Head Cluster. If this isnt created on the Cluster Master role, you wont be able to see the SHC members. In this case, on the DMC instance, you will need to add each SHC member as a search peer. After you do that, it will be visible in the DMC and you can assign the SH role to it.

[transtrophe]

So I added a comment to the thread before your response (and thanks for the response - btw) that seems to confirm that my 3 shc members are "pointing" to my index cluster master.

Regarding your obs about the DMC, that is configured on the index cluster master which is where I am observing that the Overview panel "flashes" the Search Cluster panel but then it looses visibility.On the index cluster master instance DMC I also see that the 3 SHC members show up in the Search Head tab of the Distributed Environment>Indexer Cluster panel; however, the index cluster master node is also listed in this Search Head panel. I don't think I want that and maybe that is why the index cluster master node DMC is acting the way I describe (not showing the Search Head info in the Overview panel).

[transtrophe]

I also see that in the DMC of the index cluster master node in the Distributed Environment > Distributed Search panel there are entries for the 5 index cluster peers. I don't think I should have those configured on the index cluster master in this section of the master's DMC. Maybe this is what is causing the master node instance to show up under the Search Head tab of the Distributed Environment>Indexer Cluster panel.

I could use some clarification on this before I delete those entries from the Distributed Search panel.

[transtrophe]

I found an entry in documentation that indicated the index cluster master node also will appear in the list of entries in the Search Head tab of the Distributed Environment>Indexer Cluster panel.

I don't have any further issues with this thread. Thanks to everyone that responded with guidance.

[transtrophe]

Also, here is my distsearch.conf from my index cluster master node instance:

[distributedSearch:dmc_group_license_master]

[distributedSearch:dmc_group_search_head]

[distributedSearch:dmc_group_kv_store]

[distributedSearch:dmc_group_deployment_server]

[distributedSearch:dmc_group_cluster_master]
servers = localhost:localhost

[distributedSearch:dmc_group_indexer]
default = true
servers = hostname_idx1:8089,hostname_idx2:8089,hostname_idx3:8089,hostname_idx4:8089,hostname_idx5:8089

[distributedSearch]
disabled = 0

---

Do or should I have any entries in the [distributedSearch:dmc_group_search_head] stanza?

[esix_splunk]

Additionally, all nodes not reporting in as an Indexer will report as a Search Peer in the Cluster Master -> Cluster Management view. This is expected behaviour. The delineation of role / function will occur in the DMC.

[esix_splunk]

The options for distributed search shouldnt be modified by hand. Use the DMC gui to edit this. These are assigned based on the designated roles in the DMC.