Deployment Architecture

What is the best order to perform an indexer cluster migration and expansion of both Search Head cluster and Indexer cluster?

lawannapage
New Member

What is the best order to perform the above? Our current Splunk environment consists of 5 clustered Indexers and 4 clustered Search Heads load balanced, both single-site. We will be adding 11 Indexers equally distributed between two sites, and an additional 4 Search Heads, equally distributed between two sites, all utilizing an existing Load Balancer.
Indexer cluster will become multisite, but remain single cluster. Search Head cluster will remain a logical single site cluster.

My issue is the order in which the migration and expansion steps should be done. Data migration should not be an issue; we do not currently have an official data retention policy; I'm fairly certain we will let existing single-site data age out. Here is my current outline:

Indexer expansion and Migration
1. Install and configure additional 11 Indexers
2. Add indexers to current site
3. Migrate from single-site to multisite

Search Head expansion
1. Install and configure additional 4 Search Heads
2. Add Search Heads to existing cluster
3. Add Search Heads to Load Balancer

Please advise if this is the best way to proceed, and if not please give recommendations.

0 Karma
1 Solution

adonio
Ultra Champion

Hello there
here is my recommendation based on Splunk docs and little experience:
follow this article to migrate to multi-site indexer cluster:
http://docs.splunk.com/Documentation/Splunk/6.6.2/Indexer/Migratetomultisite
general steps are:
install splunk in 11 indexers and wait with configuration as slaves
modify the master and restart
enable maintenance mode
alter indexers in the current cluster and assign site.
add the new 11 indexers to the multi-site cluster and assign the right site to them

now for the search head cluster.
follow the steps mentioned here:
https://docs.splunk.com/Documentation/Splunk/6.6.1/DistSearch/Addaclustermember
be carefull as the search heads need to be search peers too the Indexer Cluster Master!
if this configuration is deployed vie deployer (app), you supposed to be good. if it was configured via cli or directly edited server.conf in /etc/system/local/ you will have to do the same on new members

quick note on your search head cluster members distribution. it is recomended to have an odd number of search heads in each site so in case there is a site failure, the remaining members can elect a captain.
read more here:
http://docs.splunk.com/Documentation/Splunk/6.6.1/DistSearch/DeploymultisiteSHC#Important_considerat...
in your case i will reccomend a 3-5 distribution and not 6-2 as you mention in your question.

hope it helps and good luck!

View solution in original post

0 Karma

adonio
Ultra Champion

Hello there
here is my recommendation based on Splunk docs and little experience:
follow this article to migrate to multi-site indexer cluster:
http://docs.splunk.com/Documentation/Splunk/6.6.2/Indexer/Migratetomultisite
general steps are:
install splunk in 11 indexers and wait with configuration as slaves
modify the master and restart
enable maintenance mode
alter indexers in the current cluster and assign site.
add the new 11 indexers to the multi-site cluster and assign the right site to them

now for the search head cluster.
follow the steps mentioned here:
https://docs.splunk.com/Documentation/Splunk/6.6.1/DistSearch/Addaclustermember
be carefull as the search heads need to be search peers too the Indexer Cluster Master!
if this configuration is deployed vie deployer (app), you supposed to be good. if it was configured via cli or directly edited server.conf in /etc/system/local/ you will have to do the same on new members

quick note on your search head cluster members distribution. it is recomended to have an odd number of search heads in each site so in case there is a site failure, the remaining members can elect a captain.
read more here:
http://docs.splunk.com/Documentation/Splunk/6.6.1/DistSearch/DeploymultisiteSHC#Important_considerat...
in your case i will reccomend a 3-5 distribution and not 6-2 as you mention in your question.

hope it helps and good luck!

0 Karma

lawannapage
New Member

I would like clarification on this step: alter indexers in the current cluster and assign site. Do you mean to assign the new 11 indexers in the current cluster, then partition all indexers into their respective sites?

0 Karma

adonio
Ultra Champion

the new indexers are to be configured as part of a multi-site cluster hence have the site=siteNumber in server.conf

0 Karma

lawannapage
New Member

Thank you, Adonio! If anyone else has something to add, I would love to hear it.

0 Karma

lawannapage
New Member

Forgot to add that we are currently using Splunk version 6.5.2.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...