Migrate ES standalone to Cluster

[edwardrose]

I have a couple of questions about migrating the ES standalone search head to a clustered search head.  I have tested two different methods and one works well and the Splunk method doesn’t work.  Here is my method.

My Method:

1. Create tarball backups of /opt/splunk/etc/apps and /opt/splunk/etc/users from the standalone ES search head
2. Create a kvstore backup from standalone ES backup
3. Build deployer and search head cluster
1. Setup ldap
2. Setup licensing
4. Extract users.tar in /opt/splunk/etc/shcluster on deployer
5. Extract apps.tar in /opt/splunk/etc/shcluster on deployer
1. Remove all apps that are default install apps
6. Apply shcluster bundle
7. Restore kvstore on captain
8. Restore store kvstore on other two nodes in the cluster
9. Restore entire sh cluster
10. Done
1. Team tested and everything looks like it is there and seems to be working

Splunk’s Method:

1. Create tarball backups of /opt/splunk/etc/apps and /opt/splunk/etc/users from the standalone ES search head
2. Create kvstore backup from standalone ES
3. Build out deployer and search head cluster
1. Setup ldap
2. Setup licensing
4. Install latest version ES on deployer
1. Install all TA that will be used
5. Deploy ES out to cluster
6. Restore users.tar in /opt/splunk/etc/shcluster
7. Restore other apps in /opt/splunk/etc/shcluster
1. This process is more tedious as have to break out all items from ES app individually
8. Deploy apps and users out to shcluster
9. Restore kvstore on captain
10. Restore kvstore on the other two nodes
11. Restart sh cluster
12. Done
1. Team tested and it seems to be working but none of the datamodels are working and does not appear to recognize the kvstore restore.

[n0psl1de]

@bolaojewale I’m curious about those points as well. Did you find an answer? 

[bolaojewale]

Hey @n0psl1de ,

I ended up running some experiments  and formulating my own answers.

How did you migrate accelerated data models from the standalone SH to the cluster? 

• So with Splunk 8, you can share DMA. I started down this route but abandoned. I was able to create new accelerations with the SH cluster while still accelerating data models with the stand alone search head. There is documentation on DMA - https://docs.splunk.com/Documentation/Splunk/8.1.1/Knowledge/Sharedatamodelsummaries 

What was your process to your cutover from the standalone SH to the cluster?

• To complete cut over, I disabled writing to apps on the standalone to prevent users from making changes. Copied over all of the apps minus Splunk default apps. Copied over the KV store. Had users verify. That's mainly it.

[n0psl1de]

@bolaojewale Ok good to know. Thanks for the quick response. 

[bolaojewale]

Hey Edward. Thanks for your post. A few questions...

How did you migrate accelerated data models from the standalone SH to the cluster? 

What was your process to your cutover from the standalone SH to the cluster?