https://community.splunk.com/t5/Getting-Data-In/What-is-the-process-of-decommissioning-indexers/m-p/499776#M85179 <P>Add your new indexers, then update your forwarders to send data to the new indexers (not the old ones)<BR /> This will prevent the disk space on the old indexers causing the forwarding queues to fill up.</P> <P>Your first decomissioned idx will have to transfer (ideally) primacy of approx 33% of buckets, and once that is done, the cluster will replicate buckets to the remaining 5 indexers. <BR /> With 6TB volumes, this is likely to take several hours (possibly days depending on spec etc).</P> <P>Note - you could start by doing a data rebalance which will split the data equally between the old and new peers - whilst this would speed up each old decom, it is a very lengthy process, and would probably take longer than just running the decom process.<BR /> By the time you have decomissioned all 3 old indexers, you data should be reasonably well balanced across the new servers.</P> Fri, 31 Jan 2020 19:41:39 GMT nickhills 2020-01-31T19:41:39Z https://community.splunk.com/t5/Getting-Data-In/What-is-the-process-of-decommissioning-indexers/m-p/499773#M85176 <P>When you run the offline command permanently on an indexer.</P> <P>1) How much time does it take to reassign the data to other members in cluster<BR /> 2)Can we run offline command on three indexers at a time or do we need to wait for anything?<BR /> 3) Adding and removing members is correct order or the other way around?</P> Fri, 31 Jan 2020 17:54:27 GMT https://community.splunk.com/t5/Getting-Data-In/What-is-the-process-of-decommissioning-indexers/m-p/499773#M85176 vrmandadi 2020-01-31T17:54:27Z https://community.splunk.com/t5/Getting-Data-In/What-is-the-process-of-decommissioning-indexers/m-p/499774#M85177 <P>1.) It depends - how much data your indexer hold primary copies for, network/disk IO, cpu load etc. (It can take some time)<BR /> 2.) No - Dont do this, decom one indexer at a time.<BR /> 3.) Add new indexers first - more indexers in your cluster will speed up the decom process for the old ones, and will begin balancing buckets to the new peers as soon as the process starts. <BR /> Once you have added all your new indexers, remove the old ones one at a time.</P> Fri, 31 Jan 2020 17:58:22 GMT https://community.splunk.com/t5/Getting-Data-In/What-is-the-process-of-decommissioning-indexers/m-p/499774#M85177 nickhills 2020-01-31T17:58:22Z https://community.splunk.com/t5/Getting-Data-In/What-is-the-process-of-decommissioning-indexers/m-p/499775#M85178 <P>when you say some time like 6TB storage ,does it take days? or hours.We have a lot of issues with disk space space falling below 5000MB and thus the indexer stopped accepting new data.This made the HF queues getting filled.All of sudden the data burts in , data delay ...Does this cause data loss even if other Indexers are running?</P> Fri, 31 Jan 2020 18:45:56 GMT https://community.splunk.com/t5/Getting-Data-In/What-is-the-process-of-decommissioning-indexers/m-p/499775#M85178 vrmandadi 2020-01-31T18:45:56Z https://community.splunk.com/t5/Getting-Data-In/What-is-the-process-of-decommissioning-indexers/m-p/499776#M85179 <P>Add your new indexers, then update your forwarders to send data to the new indexers (not the old ones)<BR /> This will prevent the disk space on the old indexers causing the forwarding queues to fill up.</P> <P>Your first decomissioned idx will have to transfer (ideally) primacy of approx 33% of buckets, and once that is done, the cluster will replicate buckets to the remaining 5 indexers. <BR /> With 6TB volumes, this is likely to take several hours (possibly days depending on spec etc).</P> <P>Note - you could start by doing a data rebalance which will split the data equally between the old and new peers - whilst this would speed up each old decom, it is a very lengthy process, and would probably take longer than just running the decom process.<BR /> By the time you have decomissioned all 3 old indexers, you data should be reasonably well balanced across the new servers.</P> Fri, 31 Jan 2020 19:41:39 GMT https://community.splunk.com/t5/Getting-Data-In/What-is-the-process-of-decommissioning-indexers/m-p/499776#M85179 nickhills 2020-01-31T19:41:39Z https://community.splunk.com/t5/Getting-Data-In/What-is-the-process-of-decommissioning-indexers/m-p/499777#M85180 <P>Thank you </P> Fri, 31 Jan 2020 21:21:11 GMT https://community.splunk.com/t5/Getting-Data-In/What-is-the-process-of-decommissioning-indexers/m-p/499777#M85180 vrmandadi 2020-01-31T21:21:11Z https://community.splunk.com/t5/Getting-Data-In/What-is-the-process-of-decommissioning-indexers/m-p/499778#M85181 <P>@nickhillscpl how do we know that that a indexer which is decommissioning has transferred all its data to the other cluster members</P> Sat, 01 Feb 2020 02:18:54 GMT https://community.splunk.com/t5/Getting-Data-In/What-is-the-process-of-decommissioning-indexers/m-p/499778#M85181 vrmandadi 2020-02-01T02:18:54Z