Archive
Highlighted

ERROR DistributedBundleReplicationManager - got non-200 response from peer.

Explorer

Hi,

Currently, i have upgraded splunk from 6.0.4 to 6.1.1 in our test box.
Till then, i am able too the follwoig error log in splunkd.log

ERROR DistributedBundleReplicationManager - got non-200 response from peer.uri=****,
reply="HTTP/1.1 400 Bad Request" response_code=400

Could someone help to clarify and resolve the above?

Thanks
Jerina

Tags (1)
Highlighted

Re: ERROR DistributedBundleReplicationManager - got non-200 response from peer.

Explorer

I got these on old hardware when I upgraded to 6.1.3. It appears to be a timing issue and storage speed appears to play a role. Take a look at this thread.

http://answers.splunk.com/answers/12666/42-search-head-asynchronous-bundle-replication-error

0 Karma
Highlighted

Re: ERROR DistributedBundleReplicationManager - got non-200 response from peer.

Splunk Employee
Splunk Employee

This happens when the search-head is pushing a search bundle that is too large to the indexers.

The default bundle max size (maxBundleSize) is 1GB
and the default http packet size (maxcontentlength) accepted by splunkd is 800MB 😞

Therefore :

  • when 1024MB> bundle >800MB see an http error from the indexers. "failedbecauseBUNDLEDATATRANSMIT_FAILURE" or "ERROR DistributedBundleReplicationManager - got non-200 response from peer"
  • when the bundle is >1024MB we see a different error, from the search-head.

Workarounds :

  • RECOMMENDED :reduce the bundle size (trim your lookups, use blacklists in distsearch.conf)
  • LESS RECOMMENDED : allow larger bundles

example : to bump the bundle size to 2GB max
on Indexers , edit server.conf (push from cluster master etc/master-apps in a cluster)

[httpServer]
max_content_length = 2147483648 
# in bytes => 2GBdistsearch.conf 

on Search-head

[replicationSettings] 
maxBundleSize= 2097152 
# in MB => 2GB