Bundle Replication: Problem replicating config (bundle) to search peer

[sarvesh_11]

Hi,
I am very frequently getting below warning on UI:
"Bundle Replication: Problem replicating config (bundle) to search peer 'servername:8089', HTTP response code 413 (HTTP/1.1 413 Content-Length of 3174266880 too large (maximum is 3145728000)). Content-Length of 3174266880 too large (maximum is 3145728000) (Unknown write error)"

With a bit of finding and analysis, i have increased max_content_length under [http server] in $SPLUNK_HOME/etc/system/default/server.conf from 2GB to 8GB.
Also *.bundle.info files were getting coagulated at $SPLUNK_HOME/var/run/, because the size of bundle was 3.01GB and what i have set in distsearch.conf (maxbundlesize) is 3gb, so i have increased that size to 4gb.

But still my splunk directory is wholly consumed, 100% utilization.

Kindly suggest if you have any comments on this please.

[FrankVl]

What version of Splunk are you running? There have been some specific versions with bugs related to this.

Have you tried manually cleaning up that folder?

Have you tried shrinking your bundle? Do you have some very large lookups in there or so? Do you really need those on the indexers (you do if you want to use them in automatic lookups)? If not, blacklist them. Also, avoid including apps with big binary components in them, that are of no use on SH/IDX layers (e.g. scripts/binaries used for data input on a HF or so).

[sarvesh_11]

Hey @FrankVl thanks for dropping by,
This is Splunk Enterprise 6.6.3.

Our search head is Standalone, on-prem, and indexer is on cloud.

I am just wondering does max_content_length is set to 3145728000 on Indexer?

I am about to reduce the bundle size, but if i have set
max_content_lenght=8Gb in server.conf &
maxbundlesize = 4gb in distsearch.conf.

Then ideally these bundles should parse from Search Head to Indexer. And should not give error as maximum is 3145728000.

[FrankVl]

Not sure if I get your question correct, but yes, I do believe you need to set that max_content_lenght on the indexers. It is the indexers that are rejecting the oversized bundle coming from the Search Heads.

You might also want to double check using btool, that those settings are taken correctly.