What is the best order to perform an indexer cluster migration and expansion of both Search Head cluster and Indexer cluster?

[lawannapage]

What is the best order to perform the above? Our current Splunk environment consists of 5 clustered Indexers and 4 clustered Search Heads load balanced, both single-site. We will be adding 11 Indexers equally distributed between two sites, and an additional 4 Search Heads, equally distributed between two sites, all utilizing an existing Load Balancer.
Indexer cluster will become multisite, but remain single cluster. Search Head cluster will remain a logical single site cluster.

My issue is the order in which the migration and expansion steps should be done. Data migration should not be an issue; we do not currently have an official data retention policy; I'm fairly certain we will let existing single-site data age out. Here is my current outline:

Indexer expansion and Migration
1. Install and configure additional 11 Indexers
2. Add indexers to current site
3. Migrate from single-site to multisite

Search Head expansion
1. Install and configure additional 4 Search Heads
2. Add Search Heads to existing cluster
3. Add Search Heads to Load Balancer

Please advise if this is the best way to proceed, and if not please give recommendations.

[adonio · Accepted solution]

Hello there
here is my recommendation based on Splunk docs and little experience:
follow this article to migrate to multi-site indexer cluster:
http://docs.splunk.com/Documentation/Splunk/6.6.2/Indexer/Migratetomultisite
general steps are:
install splunk in 11 indexers and wait with configuration as slaves
modify the master and restart
enable maintenance mode
alter indexers in the current cluster and assign site.
add the new 11 indexers to the multi-site cluster and assign the right site to them

now for the search head cluster.
follow the steps mentioned here:
https://docs.splunk.com/Documentation/Splunk/6.6.1/DistSearch/Addaclustermember
be carefull as the search heads need to be search peers too the Indexer Cluster Master!
if this configuration is deployed vie deployer (app), you supposed to be good. if it was configured via cli or directly edited server.conf in /etc/system/local/ you will have to do the same on new members

quick note on your search head cluster members distribution. it is recomended to have an odd number of search heads in each site so in case there is a site failure, the remaining members can elect a captain.
read more here:
http://docs.splunk.com/Documentation/Splunk/6.6.1/DistSearch/DeploymultisiteSHC#Important_considerat...
in your case i will reccomend a 3-5 distribution and not 6-2 as you mention in your question.

hope it helps and good luck!

[lawannapage]

I would like clarification on this step: alter indexers in the current cluster and assign site. Do you mean to assign the new 11 indexers in the current cluster, then partition all indexers into their respective sites?

[adonio]

the new indexers are to be configured as part of a multi-site cluster hence have the site=siteNumber in server.conf

[lawannapage]

Thank you, Adonio! If anyone else has something to add, I would love to hear it.

[lawannapage]

Forgot to add that we are currently using Splunk version 6.5.2.