Deployment Architecture

Bundle Replication: Problem replicating config (bundle) to search peer

sarvesh_11
Communicator

Hi,
I am very frequently getting below warning on UI:
"Bundle Replication: Problem replicating config (bundle) to search peer 'servername:8089', HTTP response code 413 (HTTP/1.1 413 Content-Length of 3174266880 too large (maximum is 3145728000)). Content-Length of 3174266880 too large (maximum is 3145728000) (Unknown write error)"

With a bit of finding and analysis, i have increased max_content_length under [http server] in $SPLUNK_HOME/etc/system/default/server.conf from 2GB to 8GB.
Also *.bundle.info files were getting coagulated at $SPLUNK_HOME/var/run/, because the size of bundle was 3.01GB and what i have set in distsearch.conf (maxbundlesize) is 3gb, so i have increased that size to 4gb.

But still my splunk directory is wholly consumed, 100% utilization.

Kindly suggest if you have any comments on this please.

FrankVl
Ultra Champion

What version of Splunk are you running? There have been some specific versions with bugs related to this.

Have you tried manually cleaning up that folder?

Have you tried shrinking your bundle? Do you have some very large lookups in there or so? Do you really need those on the indexers (you do if you want to use them in automatic lookups)? If not, blacklist them. Also, avoid including apps with big binary components in them, that are of no use on SH/IDX layers (e.g. scripts/binaries used for data input on a HF or so).

0 Karma

sarvesh_11
Communicator

Hey @FrankVl thanks for dropping by,
This is Splunk Enterprise 6.6.3.

Our search head is Standalone, on-prem, and indexer is on cloud.

I am just wondering does max_content_length is set to 3145728000 on Indexer?

I am about to reduce the bundle size, but if i have set
max_content_lenght=8Gb in server.conf &
maxbundlesize = 4gb in distsearch.conf.

Then ideally these bundles should parse from Search Head to Indexer. And should not give error as maximum is 3145728000.

0 Karma

FrankVl
Ultra Champion

Not sure if I get your question correct, but yes, I do believe you need to set that max_content_lenght on the indexers. It is the indexers that are rejecting the oversized bundle coming from the Search Heads.

You might also want to double check using btool, that those settings are taken correctly.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk Cloud Platform 9.1.2308?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2308! Analysts can ...

Index This | Why do they call it hyper text?

November 2023 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

State of Splunk Careers 2023: Career Resilience and the Continued Value of Splunk

For the past three years, Splunk has partnered with Enterprise Strategy Group to conduct a survey that gauges ...