Deployment Architecture
Highlighted

How to migrate a clustered indexer peer to a new hardware in a single-site cluster

Explorer

Wondering if someone has gone through a hardware migration of a clustered indexers environment. Long story short, we want to move to a new platform and abandon the current hardware due to several issues that we are having with it.

All data is local to the indexers. We are running Splunk Enterprise 6.3.3.

So this is what we would like to do:
- Start a rsync from a peer A to the new host B, warm and cold buckets (peer A online)
- Either:
I)- Put CM on maintenance mode, to avoid extra replication work since the new node will have the data?
- stop peer A (and remove later after having the new peer joined with peer A data; disable maintenance mode)
OR
II-)- Remove peer A from cluster:
- splunk offline --enforce-counts
- Final rsync job to sync the latest changes on hot and cold volumes from A to host B
- Have a copy of the clustered indexes.conf loaded to the etc/system/local/ of host B before starting it
- Install same version of splunk and start splunk on host B
- Add host B to the cluster and restart host B
- Push new outputs.conf to all UFs and HFs with the new addresses of the peer B, and removing peer A
- If all goes well, then we permanently remove the old peer A:
- splunk remove cluster-peers -peers
- Repeat until all peers are removed from Cluster and moved to the new hosts.

Do you guys think this can work? Any suggestions / recommendations with this draft plan, specially regarding the options I and II above?

Thanks a lot!

0 Karma
Highlighted

Re: How to migrate a clustered indexer peer to a new hardware in a single-site cluster

Contributor

We plan the same, but our idea is to add the new nodes to the cluster and then remove node after node, giving the cluster just enough time so it can redistribute the buckets.

0 Karma
Highlighted

Re: How to migrate a clustered indexer peer to a new hardware in a single-site cluster

Explorer

Yes that's a good plan too. We are still considering do a one peer migration following the plan above and if that doesn't work well we may add all peers at once and start shutting down the old peers with the CM. Thanks for sharing.

0 Karma
Highlighted

Re: How to migrate a clustered indexer peer to a new hardware in a single-site cluster

SplunkTrust
SplunkTrust

The preferred procedure is
1) Install Splunk on the new hardware and configure it to match the old indexers
2) Add the new indexers to the cluster
3) Put the old indexers into Detention (not an option in 6.3.3)
4) Issue a 'splunk offline --enforce-counts' command to ONE old indexer
5) Wait for the buckets to migrate off the old indexer. Depending on the number of buckets, this could take a while.
6) Repeat steps 4-5 for the remaining old indexers.
7) Once all buckets are moved to the new indexers you can remove the old indexers from the cluster.

Note there is no need for rsync. The cluster automatically copies data between indexers.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

Highlighted

Re: How to migrate a clustered indexer peer to a new hardware in a single-site cluster

Contributor

Hi great answer, just a small follow-up.

What happens if the indexer was convertet to a cluster-member, will this approach work for the non-clustered data on the indexer?

Kind regards
Lars

Highlighted

Re: How to migrate a clustered indexer peer to a new hardware in a single-site cluster

Ultra Champion

No, non clustered indexes are not replicated - even with enforce-counts.

You would have to manually move those buckets onto a new indexer.

  • I did hear a rumor once that PS had a magic script that could migrate standalone buckets into clustered ones, but it was fraught with difficulties.
0 Karma