Deployment Architecture
Highlighted

How to integrate search head cluster into indexer cluster?

Communicator

I am trying to integrate a search head cluster into an indexer cluster. Search head cluster has 3 shc members using search factor = 3. Indexer cluster has 5 indexers serviced by the master with replication factor = 5. SHC captain is properly elected. DMC viewed on the Splunk Web master instance shows the index cluster deployment is functioning properly (viewed in Overview panel), but no Search Head panel is visible (although when launching Overview the Search Head cluster panel presents for a quick moment then disappears).

How can I confirm that the search head cluster got integrated to the indexer cluster? Which configuration file(s) get written when executing the edit cluster-config command?

Highlighted

Re: How to integrate search head cluster into indexer cluster?

Splunk Employee
Splunk Employee

SHC and Index Clustering are two different components in the Splunk world that do not integrate in relation to sharing functionality.

In order to "integrate" your SHC, I am guessing you mean to include Search Functionality, and perhap Search site affinity?

In the case of Search Functionality, you have to add the search head to the cluster as search peers. Additionally, you can add site member should (via server.conf,) and this will designate the search affinity for the SHC member.

Short of it, add the SHC members to the cluster as search peers. Once done, you should be able to search across the cluster and additionally see the SHC members on the CLuster Master, under search peers.

Additional configuration relating to search affinity can be found here: http://docs.splunk.com/Documentation/Splunk/6.2.2/Indexer/Multisitesearchaffinity

Regarding DMC, this cannot reside on the SHC or its members. You need to create this role outside of the Search Head Cluster. If this isnt created on the Cluster Master role, you wont be able to see the SHC members. In this case, on the DMC instance, you will need to add each SHC member as a search peer. After you do that, it will be visible in the DMC and you can assign the SH role to it.

View solution in original post

Highlighted

Re: How to integrate search head cluster into indexer cluster?

Communicator

So I added a comment to the thread before your response (and thanks for the response - btw) that seems to confirm that my 3 shc members are "pointing" to my index cluster master.

Regarding your obs about the DMC, that is configured on the index cluster master which is where I am observing that the Overview panel "flashes" the Search Cluster panel but then it looses visibility.On the index cluster master instance DMC I also see that the 3 SHC members show up in the Search Head tab of the Distributed Environment>Indexer Cluster panel; however, the index cluster master node is also listed in this Search Head panel. I don't think I want that and maybe that is why the index cluster master node DMC is acting the way I describe (not showing the Search Head info in the Overview panel).

0 Karma
Highlighted

Re: How to integrate search head cluster into indexer cluster?

Communicator

Also, here is my distsearch.conf from my index cluster master node instance:

[distributedSearch:dmcgrouplicense_master]

[distributedSearch:dmcgroupsearch_head]

[distributedSearch:dmcgroupkv_store]

[distributedSearch:dmcgroupdeployment_server]

[distributedSearch:dmcgroupcluster_master]
servers = localhost:localhost

[distributedSearch:dmcgroupindexer]
default = true
servers = hostnameidx1:8089,hostnameidx2:8089,hostnameidx3:8089,hostnameidx4:8089,hostname_idx5:8089

[distributedSearch]
disabled = 0


Do or should I have any entries in the [distributedSearch:dmcgroupsearch_head] stanza?

0 Karma
Highlighted

Re: How to integrate search head cluster into indexer cluster?

Splunk Employee
Splunk Employee

The options for distributed search shouldnt be modified by hand. Use the DMC gui to edit this. These are assigned based on the designated roles in the DMC.

0 Karma
Highlighted

Re: How to integrate search head cluster into indexer cluster?

Splunk Employee
Splunk Employee

Additionally, all nodes not reporting in as an Indexer will report as a Search Peer in the Cluster Master -> Cluster Management view. This is expected behaviour. The delineation of role / function will occur in the DMC.

0 Karma
Highlighted

Re: How to integrate search head cluster into indexer cluster?

Communicator

I also see that in the DMC of the index cluster master node in the Distributed Environment > Distributed Search panel there are entries for the 5 index cluster peers. I don't think I should have those configured on the index cluster master in this section of the master's DMC. Maybe this is what is causing the master node instance to show up under the Search Head tab of the Distributed Environment>Indexer Cluster panel.

I could use some clarification on this before I delete those entries from the Distributed Search panel.

0 Karma
Highlighted

Re: How to integrate search head cluster into indexer cluster?

Communicator

I found an entry in documentation that indicated the index cluster master node also will appear in the list of entries in the Search Head tab of the Distributed Environment>Indexer Cluster panel.

I don't have any further issues with this thread. Thanks to everyone that responded with guidance.

0 Karma
Highlighted

Re: How to integrate search head cluster into indexer cluster?

Communicator

Additionally, here are the configurations of the server.conf files on each of the shc members (showing just one instance with "sanitized values" for hostnames and secrets:

root@xxx:/home/admin# cat /opt/splunk/etc/system/local/server.conf
[sslConfig]
sslKeysfilePassword = xxxxxxxxxxxx

[lmpool:autogeneratedpooldownload-trial]
description = auto
generatedpooldownload-trial
quota = MAX
slaves = *
stack_id = download-trial

[lmpool:autogeneratedpoolforwarder]
description = auto
generatedpoolforwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:autogeneratedpoolfree]
description = auto
generatedpoolfree
quota = MAX
slaves = *
stack_id = free

[general]
pass4SymmKey = xxxxxxx
serverName = shcmember1hostname

[replication_port://9997]

[shclustering]
confdeployfetchurl = https://shcdeployerhostname
disabled = 0
mgmt
uri = https://shc_member1_hostname
pass4SymmKey = xxxxxxxxx
id = 43CD9A09-DAB9-44DF-BF43-55C0A72324FD

[clustering]
masteruri = https://masterindexer_hostname
mode = searchhead
pass4SymmKey = xxxxxxxxxx

0 Karma
Highlighted

Re: How to integrate search head cluster into indexer cluster?

Splunk Employee
Splunk Employee

Here's a topic that describes how to integrate a search head cluster with an indexer cluster:

http://docs.splunk.com/Documentation/Splunk/6.2.2/DistSearch/SHCandindexercluster