Configure one search head to search multiple clusters each with their own cluster master

[dolxor]

Hey!

If I setup two separate splunk installation in each datacenter. Logg- or machine data produced inside the datacenter (from proxies, network, applications++) will be sent to the splunk installation (we call it a "farm") in the same datacenter.

Now, I want to search data from both (or more) splunk farms in one search-head. So is it possible to;

• Use searchheads in each farm to search multiple clusters each with their own cluster master?
• Allow a "top" searchhead to search multiple clusters each with their own cluster master?
• Allow a "top" searchhead to search multiple searchheads that reside in their own clusters?

So the goal is; given clusters in each datacenter, can I search all data (from all datacenters) in one search head?

Visio image #1

Visio Image #2

[RicoSuave · Accepted solution]

First off, thanks for bringing this up in the channel today. Excellent question! Contrary to our beliefs, the answer is YES. YAY. To configure this you'll have to reference server.conf http://docs.splunk.com/Documentation/Splunk/5.0/admin/Serverconf

Straight from the docs:

##########################################################################################################
# Set this node to be a searchhead to multiple cluster masters - 
# "SplunkMaster01" with pass4SymmKey set to 'someSecret and "SplunkMaster02" with no pass4SymKey set here.
##########################################################################################################
[clustering]
mode = searchhead
master_uri = clustermaster:east, clustermaster:west

[clustermaster:east]
master_uri=https://SplunkMaster01.example.com:8089
pass4SymmKey=someSecret

[clustermaster:west]
master_uri=https://SplunkMaster02.example.com:8089

I would, however, caution you that, of course, if you have duplicate data in both datacenters, then you will end up with duplicates in your search. Simple workaround is to add dedup _raw to the base of your searches.

Additionally, depending on where your search head resides, you might run into latency issues if one cluster takes significantly longer to stream results back to the search head than the other.

[dolxor]

JoeTron, awesome answer! I have yet to familiarize myself with all the new configurations of the different .conf files in Splunk 5.0. This is exactly what I need!
Thanks for quick response both here and on Splunk's IRC channel (EFNet #Splunk)!