Deployment Architecture

Bundle replication fails with: response_code=204

drrushi_splunk
Splunk Employee
Splunk Employee

The search-head fails to retrieve results from some/all search-peers and emits messages like so on the UI:

"Problem replicating config (bundle) to search peer 'peer_host:8089', got http response code 204 HTTP/1.1 204 No Content"

The search-head splunkd.log shows:

ERROR DistributedBundleReplicationManager - got non-200 response from peer.uri=https://peer_host:8089, reply="HTTP/1.1 204 No Content" response_code=204

1 Solution

drrushi_splunk
Splunk Employee
Splunk Employee

First check the peer's splunkd.log for any messages during the same time as the search-head's DistributedBundleReplicationManager error.

If you do find in the peer's splunkd.log messages such as:

ERROR DistBundleRestHandler - File users/xxx/yyy/local/props.conf in knowledge bundle is either not in white list or else excluded by black list. Bundle /opt/splunk/var/run/searchpeers/ will be removed

...then this means that there must be on the peer a rouge 'distsearch.conf' which does't not explicitly whitelist or blacklist any bundle files ... as a result by default the peer simply rejects the bundle.

To workaround this please remove any distsearch.conf (from system/local OR etc/apps/appname/local) on the peers and restart Splunk.

In version 6.1 a new functionality was added to the peer which allows peers to blacklist/whitelist bundle contents based on locally defined rules (via local distsearch.conf).

View solution in original post

drrushi_splunk
Splunk Employee
Splunk Employee

First check the peer's splunkd.log for any messages during the same time as the search-head's DistributedBundleReplicationManager error.

If you do find in the peer's splunkd.log messages such as:

ERROR DistBundleRestHandler - File users/xxx/yyy/local/props.conf in knowledge bundle is either not in white list or else excluded by black list. Bundle /opt/splunk/var/run/searchpeers/ will be removed

...then this means that there must be on the peer a rouge 'distsearch.conf' which does't not explicitly whitelist or blacklist any bundle files ... as a result by default the peer simply rejects the bundle.

To workaround this please remove any distsearch.conf (from system/local OR etc/apps/appname/local) on the peers and restart Splunk.

In version 6.1 a new functionality was added to the peer which allows peers to blacklist/whitelist bundle contents based on locally defined rules (via local distsearch.conf).

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...