This happens when the search-head is pushing a search bundle that is too large to the indexers.
The default bundle max size (maxBundleSize) is 1GB
and the default http packet size (max_content_length) accepted by splunkd is 800MB 😞
Therefore :
when 1024MB> bundle >800MB see an http error from the indexers. "failed_because_BUNDLE_DATA_TRANSMIT_FAILURE" or "ERROR DistributedBundleReplicationManager - got non-200 response from peer"
when the bundle is >1024MB we see a different error, from the search-head.
Workarounds :
RECOMMENDED :reduce the bundle size (trim your lookups, use blacklists in distsearch.conf)
LESS RECOMMENDED : allow larger bundles
example : to bump the bundle size to 2GB max
on Indexers , edit server.conf (push from cluster master etc/master-apps in a cluster)
[httpServer]
max_content_length = 2147483648
# in bytes => 2GBdistsearch.conf
on Search-head
[replicationSettings]
maxBundleSize= 2097152
# in MB => 2GB
... View more