Limit output network bandwidth during peer decommissioning

edoardo_vicendo — Tue, 13 Oct 2020 16:17:57 GMT

Hello,

We are in a multi-site indexer cluster environment, and we are going to upgrade our infrastructure from 3 Indexer to 6 Indexer.

Basically we will add 6 new indexers, and decommission the 3 old indexers.

Do you know which is the speed in MBytes/s that will be reached once the data will start to be copied from the 3 old Indexers to the 6 new indexers?

We did some test in our development environment, where we have simulated a similar scenario, and it seems it started copying very fast with 170 MBytes/s peak (we have checked it with an nmon session at the source Indexer machine). In order to start the copying process we had:

added the new Indexers to the Master Node
switched the Splunk HFs to forward data to the new Indexers
run, one by one on each old Indexer, the command splunk offline --enforce-counts

Before starting the test we have read the following documentation:

https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Takeapeeroffline#Estimate_the_cluster_recovery_time_when_a_peer_gets_decommissioned

In the official documentation it is mentioned:

10GB (rawdata and/or index files) from one peer to another across a LAN takes about 5-10 minutes

therefore copying speed should go from 136 Mbit/s to 272 Mbit/s.

If the speed we have observed is correct (more than 1Gbit/s), do you know if there is any way to limit the output bandwidth?

Thanks a lot,

Edoardo

Re: Limit output network bandwidth during peer decommissioning

richgalloway — Tue, 13 Oct 2020 17:03:42 GMT

Check out the max_nonhot_rep_kBps setting in server.conf.

Re: Limit output network bandwidth during peer decommissioning

edoardo_vicendo — Thu, 15 Oct 2020 15:15:07 GMT

@richgalloway Thanks a lot for your suggestion!

Tested and it works perfectly, and you can even change it in the meanwhile:

splunk edit cluster-config -max_nonhot_rep_kBps <value>

topic Re: Limit output network bandwidth during peer decommissioning in Getting Data In

Limit output network bandwidth during peer decommissioning

Re: Limit output network bandwidth during peer decommissioning

Re: Limit output network bandwidth during peer decommissioning