Deployment Architecture

Configure one search head to search multiple clusters each with their own cluster master

dolxor
Path Finder

Hey!

If I setup two separate splunk installation in each datacenter. Logg- or machine data produced inside the datacenter (from proxies, network, applications++) will be sent to the splunk installation (we call it a "farm") in the same datacenter.

Now, I want to search data from both (or more) splunk farms in one search-head. So is it possible to;

  • Use searchheads in each farm to search multiple clusters each with their own cluster master?
  • Allow a "top" searchhead to search multiple clusters each with their own cluster master?
  • Allow a "top" searchhead to search multiple searchheads that reside in their own clusters?

So the goal is; given clusters in each datacenter, can I search all data (from all datacenters) in one search head?

Visio image #1

Visio Image #2

Tags (2)
1 Solution

RicoSuave
Builder

First off, thanks for bringing this up in the channel today. Excellent question! Contrary to our beliefs, the answer is YES. YAY. To configure this you'll have to reference server.conf http://docs.splunk.com/Documentation/Splunk/5.0/admin/Serverconf

Straight from the docs:

##########################################################################################################
# Set this node to be a searchhead to multiple cluster masters - 
# "SplunkMaster01" with pass4SymmKey set to 'someSecret and "SplunkMaster02" with no pass4SymKey set here.
##########################################################################################################
[clustering]
mode = searchhead
master_uri = clustermaster:east, clustermaster:west

[clustermaster:east]
master_uri=https://SplunkMaster01.example.com:8089
pass4SymmKey=someSecret

[clustermaster:west]
master_uri=https://SplunkMaster02.example.com:8089

I would, however, caution you that, of course, if you have duplicate data in both datacenters, then you will end up with duplicates in your search. Simple workaround is to add dedup _raw to the base of your searches.

Additionally, depending on where your search head resides, you might run into latency issues if one cluster takes significantly longer to stream results back to the search head than the other.

View solution in original post

RicoSuave
Builder

First off, thanks for bringing this up in the channel today. Excellent question! Contrary to our beliefs, the answer is YES. YAY. To configure this you'll have to reference server.conf http://docs.splunk.com/Documentation/Splunk/5.0/admin/Serverconf

Straight from the docs:

##########################################################################################################
# Set this node to be a searchhead to multiple cluster masters - 
# "SplunkMaster01" with pass4SymmKey set to 'someSecret and "SplunkMaster02" with no pass4SymKey set here.
##########################################################################################################
[clustering]
mode = searchhead
master_uri = clustermaster:east, clustermaster:west

[clustermaster:east]
master_uri=https://SplunkMaster01.example.com:8089
pass4SymmKey=someSecret

[clustermaster:west]
master_uri=https://SplunkMaster02.example.com:8089

I would, however, caution you that, of course, if you have duplicate data in both datacenters, then you will end up with duplicates in your search. Simple workaround is to add dedup _raw to the base of your searches.

Additionally, depending on where your search head resides, you might run into latency issues if one cluster takes significantly longer to stream results back to the search head than the other.

dolxor
Path Finder

JoeTron, awesome answer! I have yet to familiarize myself with all the new configurations of the different .conf files in Splunk 5.0. This is exactly what I need!
Thanks for quick response both here and on Splunk's IRC channel (EFNet #Splunk)!

0 Karma
Get Updates on the Splunk Community!

New Cloud Intrusion Detection System Add-on for Splunk

In July 2022 Splunk released the Cloud IDS add-on which expanded Splunk capabilities in security and data ...

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...