Getting Data In

What is the process of decommissioning indexers?

vrmandadi
Builder

When you run the offline command permanently on an indexer.

1) How much time does it take to reassign the data to other members in cluster
2)Can we run offline command on three indexers at a time or do we need to wait for anything?
3) Adding and removing members is correct order or the other way around?

0 Karma
1 Solution

nickhills
Ultra Champion

1.) It depends - how much data your indexer hold primary copies for, network/disk IO, cpu load etc. (It can take some time)
2.) No - Dont do this, decom one indexer at a time.
3.) Add new indexers first - more indexers in your cluster will speed up the decom process for the old ones, and will begin balancing buckets to the new peers as soon as the process starts.
Once you have added all your new indexers, remove the old ones one at a time.

If my comment helps, please give it a thumbs up!

View solution in original post

0 Karma

nickhills
Ultra Champion

1.) It depends - how much data your indexer hold primary copies for, network/disk IO, cpu load etc. (It can take some time)
2.) No - Dont do this, decom one indexer at a time.
3.) Add new indexers first - more indexers in your cluster will speed up the decom process for the old ones, and will begin balancing buckets to the new peers as soon as the process starts.
Once you have added all your new indexers, remove the old ones one at a time.

If my comment helps, please give it a thumbs up!
0 Karma

vrmandadi
Builder

when you say some time like 6TB storage ,does it take days? or hours.We have a lot of issues with disk space space falling below 5000MB and thus the indexer stopped accepting new data.This made the HF queues getting filled.All of sudden the data burts in , data delay ...Does this cause data loss even if other Indexers are running?

0 Karma

nickhills
Ultra Champion

Add your new indexers, then update your forwarders to send data to the new indexers (not the old ones)
This will prevent the disk space on the old indexers causing the forwarding queues to fill up.

Your first decomissioned idx will have to transfer (ideally) primacy of approx 33% of buckets, and once that is done, the cluster will replicate buckets to the remaining 5 indexers.
With 6TB volumes, this is likely to take several hours (possibly days depending on spec etc).

Note - you could start by doing a data rebalance which will split the data equally between the old and new peers - whilst this would speed up each old decom, it is a very lengthy process, and would probably take longer than just running the decom process.
By the time you have decomissioned all 3 old indexers, you data should be reasonably well balanced across the new servers.

If my comment helps, please give it a thumbs up!

vrmandadi
Builder

Thank you

0 Karma

vrmandadi
Builder

@nickhillscpl how do we know that that a indexer which is decommissioning has transferred all its data to the other cluster members

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...