Bundle replication fails with: response_code=204

[drrushi_splunk]

The search-head fails to retrieve results from some/all search-peers and emits messages like so on the UI:

"Problem replicating config (bundle) to search peer 'peer_host:8089', got http response code 204 HTTP/1.1 204 No Content"

The search-head splunkd.log shows:

ERROR DistributedBundleReplicationManager - got non-200 response from peer.uri=https://peer_host:8089, reply="HTTP/1.1 204 No Content" response_code=204

[drrushi_splunk · Accepted solution]

First check the peer's splunkd.log for any messages during the same time as the search-head's DistributedBundleReplicationManager error.

If you do find in the peer's splunkd.log messages such as:

ERROR DistBundleRestHandler - File users/xxx/yyy/local/props.conf in knowledge bundle is either not in white list or else excluded by black list. Bundle /opt/splunk/var/run/searchpeers/ will be removed

...then this means that there must be on the peer a rouge 'distsearch.conf' which does't not explicitly whitelist or blacklist any bundle files ... as a result by default the peer simply rejects the bundle.

To workaround this please remove any distsearch.conf (from system/local OR etc/apps/appname/local) on the peers and restart Splunk.

In version 6.1 a new functionality was added to the peer which allows peers to blacklist/whitelist bundle contents based on locally defined rules (via local distsearch.conf).