Getting Data In

What is the process of decommissioning indexers?

vrmandadi
Builder

When you run the offline command permanently on an indexer.

1) How much time does it take to reassign the data to other members in cluster
2)Can we run offline command on three indexers at a time or do we need to wait for anything?
3) Adding and removing members is correct order or the other way around?

0 Karma
1 Solution

nickhills
Ultra Champion

1.) It depends - how much data your indexer hold primary copies for, network/disk IO, cpu load etc. (It can take some time)
2.) No - Dont do this, decom one indexer at a time.
3.) Add new indexers first - more indexers in your cluster will speed up the decom process for the old ones, and will begin balancing buckets to the new peers as soon as the process starts.
Once you have added all your new indexers, remove the old ones one at a time.

If my comment helps, please give it a thumbs up!

View solution in original post

0 Karma

nickhills
Ultra Champion

1.) It depends - how much data your indexer hold primary copies for, network/disk IO, cpu load etc. (It can take some time)
2.) No - Dont do this, decom one indexer at a time.
3.) Add new indexers first - more indexers in your cluster will speed up the decom process for the old ones, and will begin balancing buckets to the new peers as soon as the process starts.
Once you have added all your new indexers, remove the old ones one at a time.

If my comment helps, please give it a thumbs up!
0 Karma

vrmandadi
Builder

when you say some time like 6TB storage ,does it take days? or hours.We have a lot of issues with disk space space falling below 5000MB and thus the indexer stopped accepting new data.This made the HF queues getting filled.All of sudden the data burts in , data delay ...Does this cause data loss even if other Indexers are running?

0 Karma

nickhills
Ultra Champion

Add your new indexers, then update your forwarders to send data to the new indexers (not the old ones)
This will prevent the disk space on the old indexers causing the forwarding queues to fill up.

Your first decomissioned idx will have to transfer (ideally) primacy of approx 33% of buckets, and once that is done, the cluster will replicate buckets to the remaining 5 indexers.
With 6TB volumes, this is likely to take several hours (possibly days depending on spec etc).

Note - you could start by doing a data rebalance which will split the data equally between the old and new peers - whilst this would speed up each old decom, it is a very lengthy process, and would probably take longer than just running the decom process.
By the time you have decomissioned all 3 old indexers, you data should be reasonably well balanced across the new servers.

If my comment helps, please give it a thumbs up!

vrmandadi
Builder

Thank you

0 Karma

vrmandadi
Builder

@nickhillscpl how do we know that that a indexer which is decommissioning has transferred all its data to the other cluster members

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...