Getting Data In

Limit output network bandwidth during peer decommissioning

edoardo_vicendo
Builder

Hello,

We are in a multi-site indexer cluster environment, and we are going to upgrade our infrastructure from 3 Indexer to 6 Indexer.

Basically we will add 6 new indexers, and decommission the 3 old indexers.

Do you know which is the speed in MBytes/s that will be reached once the data will start to be copied from the 3 old Indexers to the 6 new indexers?

We did some test in our development environment, where we have simulated a similar scenario, and it seems it started copying very fast with 170 MBytes/s peak (we have checked it with an nmon session at the source Indexer machine). In order to start the copying process we had:

  • added the new Indexers to the Master Node
  • switched the Splunk HFs to forward data to the new Indexers
  • run, one by one on each old Indexer, the command splunk offline --enforce-counts

Before starting the test we have read the following documentation:

https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Takeapeeroffline#Estimate_the_cluster_re...

In the official documentation it is mentioned:

10GB (rawdata and/or index files) from one peer to another across a LAN takes about 5-10 minutes

therefore copying speed should go from 136 Mbit/s to 272 Mbit/s.

If the speed we have observed is correct (more than 1Gbit/s), do you know if there is any way to limit the output bandwidth?

Thanks a lot,

Edoardo

Labels (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Check out the max_nonhot_rep_kBps setting in server.conf.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Check out the max_nonhot_rep_kBps setting in server.conf.

---
If this reply helps you, Karma would be appreciated.

edoardo_vicendo
Builder

@richgalloway Thanks a lot for your suggestion!

Tested and it works perfectly, and you can even change it in the meanwhile:

splunk edit cluster-config -max_nonhot_rep_kBps <value>

 

0 Karma
Get Updates on the Splunk Community!

Splunk + ThousandEyes: Correlate frontend, app, and network data to troubleshoot ...

 Are you tired of troubleshooting delays caused by siloed frontend, application, and network data? We've got a ...

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI!Discover how Splunk’s agentic AI ...

🔐 Trust at Every Hop: How mTLS in Splunk Enterprise 10.0 Makes Security Simpler

From Idea to Implementation: Why Splunk Built mTLS into Splunk Enterprise 10.0  mTLS wasn’t just a checkbox ...