How to migrate and update clustered environment?

[giulioBalza]

Hello,

we have a cluster environment:

- Search Head Cluster (3 nodes)

- Indexers Cluster (4 sites) 10 nodes each

actually is still with version 7.3.9 based on CentOS.

We have to migrate the OS to Suse linux and at the same time upgrade to Splunk 8.2.6 ,  we want to prepare a parallel environment with the same number of nodes where to install the latest Splunk version.

We also would like to use this new environment to migrate and fix the apps to be compatible with python, xml and jquery then start the env in production.

We are struggling to find a way to migrate the indexes buckets (db_* and rb_*) and kvstore from old to new environment with less downtime and loss data, if it is possible what about the the GUID in buckets name.

Thank you.

[richgalloway]

Normally, I'd recommend adding new instances to each cluster and retiring the old ones.  You have two complicating factors, however: 1) a change in OS and 2) a change in major version.  Either of those by themselves might not be a deal-breaker since a difference in OS or Splunk version is expected during an upgrade, but changing both at the same time should be approached carefully.  If you have a dev/test system then test your upgrade procedures there.

Consider doing upgrade just the OS using new instances running the same Splunk version.  Once the clusters are moved to the new instances then upgrade Splunk.

It might be worthwhile to contact your Splunk account team about having an architect create a plan.

[PickleRick]

+1 for what @richgalloway said.

Limit what you do in each step so you always know what changed, what could be the cause of possible problems and what can be rolled back if needed.

I think I'd try to:

1) Add new nodes (still using the same splunk version) to indexer cluster, decomission old ones.

2) Decomission old cluster master and replace it with a new one (still the same splunk version, but on new OS)

3) Create a new SH cluster (still the same splunk version), migrate apps, data, kvstore to the new one, decomission old one

4) Approach the Splunk upgrade process

It's of course a rough outline since it doesn't touch deployment server, license master, monitoring console and so on. For detailed plan I'd definitelly call your friendly local partner's support and/or Splunk's PS team.

[isoutamo]

Hi

Couple of things which @richgalloway or @PickleRick  haven't mentioned yet.

You cannot do a live update from 7.3 to 8.2! You must go through (8.0 or) 8.1.x!

1. So this means one data migration with 7.3 version first like @richgalloway and @PickleRick proposed.
2. Of course you must fix those apps etc. which need to fix before go to 8.x.
3. Then after you have migrated data you can do live update from7.3 to e.g. 8.1.10.
4. Then fix what is needed (e.g. python) 
5. Live migration to 8.2.6 (or maybe better to wait 8.2.7+)

If splunk works ok with CentOS and SUSE at same time you should also migrate SHC instead of set it up from scratch. This way you avoid to deploy all local changes done on SHC nodes via Deployer (this can cause some challenges later on when users want to do some changes to e.g. alerts).

r. Ismo

[PickleRick]

Yes, I "included" it all under the last point of "upgrade splunk" without going into too much details about intermediate version, verify python compatibility and such. I simply assume that it would be taken care of by someone knowledgeable 🙂

And yes, I agree that it might indeed be easier to add new nodes to SHC and decommission old ones. Otherwise you might need together all the locally-made changes from the cluster nodes to an additional app or something like that (of course with additional caveats with the "standard" apps like search).

[giulioBalza]

Hi Rickgalloway,

thanks for the reply, sure changing OS and major version at the same moment is not an easy step.

For this reason we install a separate environment with new OS and latest Splunk version identical to the old one.

What is not clear is:

- how to copy/move buckets from old to new (guid changes, reduce downtime) 

- kvstore migration 

thanks

[richgalloway · Accepted solution]

What I'm saying is don't do any of that.

Add the new hardware with the new OS and same Splunk version to the existing cluster rather than as a parallel cluster.  That way, the cluster manages the data for you.

Once the old hardware has been replaced by new then you can upgrade Splunk.

Of course, as @PickleRick said, there's much more to it and we can go into details if you want, but I strongly recommend reaching out to Splunk.