Migrating separate environments to Search Head Clustering, what are things I should watch out for?

[sat94541]

I have a few separate environments. For some, we use a Standalone Search Head and for others, we use Search Head Pooling. We need to plan and move these to Search Head Clustering.

I am looking for very high level overview of the things to watch out for.

[Steve_G_]

sat94541 - The information you need to deploy a search head cluster and migrate settings to it from existing environments is already documented in full, in the Distributed Search manual. It's actually remarkably simple and straightforward. Be aware that the documentation contains information on requirements and such that is not included in the previous answer.

To deploy a search head cluster and migrate existing settings to it, just follow these steps:

1. Read the list of system requirements and other considerations, here: http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/SHCsystemrequirements
2. Follow the procedure for deploying the cluster, here: http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/SHCdeploymentoverview
3. Depending on whether you are migrating settings from a search head pool or from a standalone search head, see one of these two topics:
   • http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/Migratefromsearchheadpooling
   • http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/Migratefromstandalonesearchheads

Thank you.

[rbal_splunk · Accepted solution]

I have outlined very high level the steps required for Migration from search Head Pooling to a Search Head Cluster. Before you refer to this outline, please read and review all sections on Search Head Clustering at the following documentation link: http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/Whatisdistributedsearch

The outline below is only to highlight some of the additional information and it is not a replacement of documentation on Search Head Clustering. First, review the online documentation and after that, refer to the steps outlined here.
Also, my suggestion will be to test this first before applying them to Production.

i) For Search Head Clustering, you need a minimum of 3 instances of Splunk as Search Head Cluster Members and one additional instance as deployer. A Search Head Cluster Member should be fresh install of Splunk and not re-purposed splunk instance.

ii) Identify your requirements

*Determine the cluster size, that is, the number of search heads that you want to include in it.
*Decide what replication factor you want to implement. The replication factor is the number of copies of search artifacts that the cluster maintains. Your optimal replication factor depends on factors specific to your environment, but essentially involves a trade-off between failure tolerance and storage capacity. Determine whether the search head cluster will be running against a group of standalone indexers or an indexer cluster.

iii) Set up the deployer:
iv) Install the Splunk Enterprise instances that will be search head cluster Members.
v) Initialize cluster members
vi) Bring up the cluster captain by bootstrapping members.
vii) Perform following post-deployment setup

• Integrate the search head cluster into an indexer cluster (provided index replication is being used)
• Connect the search heads to their search peers.
• **Configure LDAP for User. For this step refer Answers link -- http://answers.splunk.com/answers/230771/search-head-cluster-how-to-manage-new-roles-betwee.html#ans...
• Install a load balancer in front of the search heads.
• Use the deployer to distribute apps and configuration updates to the search heads. Below is key information for these steps. Also refer to this documentation: http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/Migratefromsearchheadpooling#Migrate_to...

You can migrate the settings from a search head pool to a search head cluster or from Standalone Search Head to Search Head Cluster.

To migrate the settings from a search head pool to the search head cluster, you copy the shared directories in the search head pool to the deployer instance. The migration procedure varies somewhat depending on whether you are migrating to a new cluster or to a cluster that is already running. Detailed steps are covered in documentation:

http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/Migratefromsearchheadpooling http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/Migratefromstandalonesearchheads

😎 Other Important points to consider:

8.1) Do not migrate default apps like search App, Launcher app etc.
Now, if you have custom objects like Dashboard or Reports created in these default apps, follow the steps below to migrate these:

a. Copy the .../search/local directory in the distribution directory to a new app directory, such as search_migration_app, in the
temporary directory. Do not name this new app "search."
b. Export the settings globally to make them available to all apps, including the search app. To do this, create a .../search_migration_app/metadata/local.meta file and populate it with the following content:
[]
export=system

8.2) To migrate User Preferences you may want to refer to http://answers.splunk.com/answers/232943/how-to-deploy-ui-prefconf-in-search-head-cluster-u.html

8.3) To migrate from an existing search head or search head pool over to search head clustering, users are required to place migrated assets on the deployer and push. This means migrated assets cannot be deleted, moved, etc. In particular, private user assets cannot be shared up the app level. The impact here has turned out to be greater than expected and Splunk has a Bug open for it. Bug# SPL-98917/ SPL-99099:[SHPNEXT] to figure out a way to migrate private user assets into a search head cluster such that they can be deleted, moved, shared, etc
This Bug is fixed in Splunk version 6.2.7 (and 6.3) as mentioned in --http://docs.splunk.com/Documentation/Splunk/6.2.7/DistSearch/UpgradeaSHC#Deployer_initiates_restart_...

For the version pre 6.3 and 6.2.7 Bug SPL-99099: has a script that can be used as workaround.

9) In addition you may also like to refer to known Bugs for Search Head Clustering at http://docs.splunk.com/Documentation/Splunk/6.2.3/ReleaseNotes/Knownissues and look under section “Distributed search and search head clustering issues"

[Lucas_K]

In relation to note 8.3 "In the mean time Bug SPL-99099: has a script that can be used as workaround. In case you hit this issue, please contact Splunk Support and refer to Bug SPL-99099"

We've used the script but please note that the local account MUST exist first otherwise the script will fail. This becomes an issue with an ldap powered splunk installation where a user hasn't yet logged in (ie. prior to migration).

I made a small shell wrapper to check against ldap, stage the working accounts into a migration area then runs the support migration script against that, logs out the outcome for troubleshooting later. Link : http://pastebin.com/y1JjKhGL

Oh, the script also breaks for any users with multiline savedsearches 😞 These need to be detected prior to migration. Multiline searches are the ones that have backslash characters at the end of the line to indicate carrage returns. I only noticed this AFTER migration. They get partially migrated and require some captain/debug-refresh/manual gui intervention tom foolery to fix it. 😞

[sk314]

In addition to this (quite comprehensive) answer, please be aware of what gets replicated across search head cluster and what doesn't. In general knowledge-objects get replicated, for everything else use the deployer to maintain consistency across search heads. If you are not careful with this, you'll end up with search heads with inconsistent settings.

For reference: http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/HowconfigurationworksinSHC