Splunk architecture question

[splunkreal]

Hello,

• Could you let us know if it’s possible to connect one cluster master to another cluster indexers using distributed search or clustering settings?

Example :

testenvmgt1 (management/cluster master/shc deployer)

testenvsh1 (search head/kv) ------------------------------------> productionenvidx1/productionenvidx2 (in another cluster)
testenvsh2 (search head/kv)

testenv hasn’t any indexer.

I think we can use distributed search but I’m afraid we may get duplicate results without being in a cluster?

• Also which replication/search factor should we use (1?) as we don’t have 3 SHs as documented.

[s2_splunk · Accepted solution]

If you want to search an indexer cluster, you have to connect your SH to the corresponding Cluster Master.
There is no issue making a SH be search two (or more) separate indexer clusters; just add both cluster masters to your search head configuration. This is documented here.

[splunkreal]

Thanks a lot! So is it from each test search head to the production cluster master (management)?

[s2_splunk]

Yes, do it on every search head that needs to search your production index cluster.
For SHC, take a look here.

[splunkreal]

One last question : I have often this message "waiting for requisite number of peers to join the cluster" on the test environment as there isn't any indexer on the test cluster master (that CM will be used to deploy SH configurations/apps).

Also why my management servers (cluster masters) are listed in 'search heads' on the master dashboard?

Thanks a lot.

[traxxasbreaker]

OK, it sounds like you have a test environment with a partial search head cluster that you want to search your production indexer cluster. In that case, assuming that you have a separate cluster master for your production indexer cluster, the replication and search factor on your test environment cluster master won't do anything since it is not controlling any test indexers. The replication and search factors on your test cluster master also will not have any affect on your test search heads.

That said, you should be able to configure server.conf via the deployer on your test search heads to search your production indexer cluster, you'll just need to make sure that the plain text value of pass4SymKey matches between the two. You'd have to point it to your production cluster master because your test cluster master (hopefully) isn't controlling your production indexers.

As far as having both cluster masters control your production indexers, the indexers would only be able to point to a single cluster master to control their configurations and replication behavior. Even if they could talk to both, you wouldn't want testing in your lab to be potentially breaking things on your production indexer cluster.

[traxxasbreaker]

Yes, like that document. You would be doing it from your deployer in your lab environment within an app that you would push out to your search head cluster members. You should not add your production indexers to your test cluster master.

[splunkreal]

Hello traxxasbreaker, do you mean enabling test search heads as shown at http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Enablethesearchhead? If yes, is it from the testenv cluster master (Distributed environment/Indexer clustering/Node type/Search head node)

Or is it adding each production indexer in the test cluster master distributed search? (Distributed Environment/Distributed search)