Currently, i have upgraded splunk from 6.0.4 to 6.1.1 in our test box.
Till then, i am able too the follwoig error log in splunkd.log
ERROR DistributedBundleReplicationManager - got non-200 response from peer.uri=****,
reply="HTTP/1.1 400 Bad Request" response_code=400
Could someone help to clarify and resolve the above?
I got these on old hardware when I upgraded to 6.1.3. It appears to be a timing issue and storage speed appears to play a role. Take a look at this thread.
This happens when the search-head is pushing a search bundle that is too large to the indexers.
The default bundle max size (maxBundleSize) is 1GB
and the default http packet size (maxcontentlength) accepted by splunkd is 800MB 😞
example : to bump the bundle size to 2GB max
on Indexers , edit server.conf (push from cluster master etc/master-apps in a cluster)
[httpServer] max_content_length = 2147483648 # in bytes => 2GBdistsearch.conf
[replicationSettings] maxBundleSize= 2097152 # in MB => 2GB