Getting Data In

What is the process of decommissioning indexers?

vrmandadi
Builder

When you run the offline command permanently on an indexer.

1) How much time does it take to reassign the data to other members in cluster
2)Can we run offline command on three indexers at a time or do we need to wait for anything?
3) Adding and removing members is correct order or the other way around?

0 Karma
1 Solution

nickhills
Ultra Champion

1.) It depends - how much data your indexer hold primary copies for, network/disk IO, cpu load etc. (It can take some time)
2.) No - Dont do this, decom one indexer at a time.
3.) Add new indexers first - more indexers in your cluster will speed up the decom process for the old ones, and will begin balancing buckets to the new peers as soon as the process starts.
Once you have added all your new indexers, remove the old ones one at a time.

If my comment helps, please give it a thumbs up!

View solution in original post

0 Karma

nickhills
Ultra Champion

1.) It depends - how much data your indexer hold primary copies for, network/disk IO, cpu load etc. (It can take some time)
2.) No - Dont do this, decom one indexer at a time.
3.) Add new indexers first - more indexers in your cluster will speed up the decom process for the old ones, and will begin balancing buckets to the new peers as soon as the process starts.
Once you have added all your new indexers, remove the old ones one at a time.

If my comment helps, please give it a thumbs up!
0 Karma

vrmandadi
Builder

when you say some time like 6TB storage ,does it take days? or hours.We have a lot of issues with disk space space falling below 5000MB and thus the indexer stopped accepting new data.This made the HF queues getting filled.All of sudden the data burts in , data delay ...Does this cause data loss even if other Indexers are running?

0 Karma

nickhills
Ultra Champion

Add your new indexers, then update your forwarders to send data to the new indexers (not the old ones)
This will prevent the disk space on the old indexers causing the forwarding queues to fill up.

Your first decomissioned idx will have to transfer (ideally) primacy of approx 33% of buckets, and once that is done, the cluster will replicate buckets to the remaining 5 indexers.
With 6TB volumes, this is likely to take several hours (possibly days depending on spec etc).

Note - you could start by doing a data rebalance which will split the data equally between the old and new peers - whilst this would speed up each old decom, it is a very lengthy process, and would probably take longer than just running the decom process.
By the time you have decomissioned all 3 old indexers, you data should be reasonably well balanced across the new servers.

If my comment helps, please give it a thumbs up!

vrmandadi
Builder

Thank you

0 Karma

vrmandadi
Builder

@nickhillscpl how do we know that that a indexer which is decommissioning has transferred all its data to the other cluster members

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...