Getting Data In

What is the process of decommissioning indexers?

vrmandadi
Builder

When you run the offline command permanently on an indexer.

1) How much time does it take to reassign the data to other members in cluster
2)Can we run offline command on three indexers at a time or do we need to wait for anything?
3) Adding and removing members is correct order or the other way around?

0 Karma
1 Solution

nickhills
Ultra Champion

1.) It depends - how much data your indexer hold primary copies for, network/disk IO, cpu load etc. (It can take some time)
2.) No - Dont do this, decom one indexer at a time.
3.) Add new indexers first - more indexers in your cluster will speed up the decom process for the old ones, and will begin balancing buckets to the new peers as soon as the process starts.
Once you have added all your new indexers, remove the old ones one at a time.

If my comment helps, please give it a thumbs up!

View solution in original post

0 Karma

nickhills
Ultra Champion

1.) It depends - how much data your indexer hold primary copies for, network/disk IO, cpu load etc. (It can take some time)
2.) No - Dont do this, decom one indexer at a time.
3.) Add new indexers first - more indexers in your cluster will speed up the decom process for the old ones, and will begin balancing buckets to the new peers as soon as the process starts.
Once you have added all your new indexers, remove the old ones one at a time.

If my comment helps, please give it a thumbs up!
0 Karma

vrmandadi
Builder

when you say some time like 6TB storage ,does it take days? or hours.We have a lot of issues with disk space space falling below 5000MB and thus the indexer stopped accepting new data.This made the HF queues getting filled.All of sudden the data burts in , data delay ...Does this cause data loss even if other Indexers are running?

0 Karma

nickhills
Ultra Champion

Add your new indexers, then update your forwarders to send data to the new indexers (not the old ones)
This will prevent the disk space on the old indexers causing the forwarding queues to fill up.

Your first decomissioned idx will have to transfer (ideally) primacy of approx 33% of buckets, and once that is done, the cluster will replicate buckets to the remaining 5 indexers.
With 6TB volumes, this is likely to take several hours (possibly days depending on spec etc).

Note - you could start by doing a data rebalance which will split the data equally between the old and new peers - whilst this would speed up each old decom, it is a very lengthy process, and would probably take longer than just running the decom process.
By the time you have decomissioned all 3 old indexers, you data should be reasonably well balanced across the new servers.

If my comment helps, please give it a thumbs up!

vrmandadi
Builder

Thank you

0 Karma

vrmandadi
Builder

@nickhillscpl how do we know that that a indexer which is decommissioning has transferred all its data to the other cluster members

0 Karma
Get Updates on the Splunk Community!

Index This | Why do they call it hyper text?

November 2023 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

State of Splunk Careers 2023: Career Resilience and the Continued Value of Splunk

For the past three years, Splunk has partnered with Enterprise Strategy Group to conduct a survey that gauges ...

The Great Resilience Quest: 9th Leaderboard Update

The ninth leaderboard update (11.9-11.22) for The Great Resilience Quest is out >> Kudos to all the ...