Getting Data In

Limit output network bandwidth during peer decommissioning

edoardo_vicendo
Contributor

Hello,

We are in a multi-site indexer cluster environment, and we are going to upgrade our infrastructure from 3 Indexer to 6 Indexer.

Basically we will add 6 new indexers, and decommission the 3 old indexers.

Do you know which is the speed in MBytes/s that will be reached once the data will start to be copied from the 3 old Indexers to the 6 new indexers?

We did some test in our development environment, where we have simulated a similar scenario, and it seems it started copying very fast with 170 MBytes/s peak (we have checked it with an nmon session at the source Indexer machine). In order to start the copying process we had:

  • added the new Indexers to the Master Node
  • switched the Splunk HFs to forward data to the new Indexers
  • run, one by one on each old Indexer, the command splunk offline --enforce-counts

Before starting the test we have read the following documentation:

https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Takeapeeroffline#Estimate_the_cluster_re...

In the official documentation it is mentioned:

10GB (rawdata and/or index files) from one peer to another across a LAN takes about 5-10 minutes

therefore copying speed should go from 136 Mbit/s to 272 Mbit/s.

If the speed we have observed is correct (more than 1Gbit/s), do you know if there is any way to limit the output bandwidth?

Thanks a lot,

Edoardo

Labels (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Check out the max_nonhot_rep_kBps setting in server.conf.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Check out the max_nonhot_rep_kBps setting in server.conf.

---
If this reply helps you, Karma would be appreciated.

edoardo_vicendo
Contributor

@richgalloway Thanks a lot for your suggestion!

Tested and it works perfectly, and you can even change it in the meanwhile:

splunk edit cluster-config -max_nonhot_rep_kBps <value>

 

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...