I am very frequently getting below warning on UI:
"Bundle Replication: Problem replicating config (bundle) to search peer 'servername:8089', HTTP response code 413 (HTTP/1.1 413 Content-Length of 3174266880 too large (maximum is 3145728000)). Content-Length of 3174266880 too large (maximum is 3145728000) (Unknown write error)"
With a bit of finding and analysis, i have increased max_content_length under [http server] in $SPLUNK_HOME/etc/system/default/server.conf from 2GB to 8GB.
Also *.bundle.info files were getting coagulated at $SPLUNK_HOME/var/run/, because the size of bundle was 3.01GB and what i have set in distsearch.conf (maxbundlesize) is 3gb, so i have increased that size to 4gb.
But still my splunk directory is wholly consumed, 100% utilization.
Kindly suggest if you have any comments on this please.
What version of Splunk are you running? There have been some specific versions with bugs related to this.
Have you tried manually cleaning up that folder?
Have you tried shrinking your bundle? Do you have some very large lookups in there or so? Do you really need those on the indexers (you do if you want to use them in automatic lookups)? If not, blacklist them. Also, avoid including apps with big binary components in them, that are of no use on SH/IDX layers (e.g. scripts/binaries used for data input on a HF or so).
Not sure if I get your question correct, but yes, I do believe you need to set that max_content_lenght on the indexers. It is the indexers that are rejecting the oversized bundle coming from the Search Heads.
You might also want to double check using btool, that those settings are taken correctly.