Deployment Architecture

Bundle replication fails with: response_code=204

drrushi_splunk
Splunk Employee
Splunk Employee

The search-head fails to retrieve results from some/all search-peers and emits messages like so on the UI:

"Problem replicating config (bundle) to search peer 'peer_host:8089', got http response code 204 HTTP/1.1 204 No Content"

The search-head splunkd.log shows:

ERROR DistributedBundleReplicationManager - got non-200 response from peer.uri=https://peer_host:8089, reply="HTTP/1.1 204 No Content" response_code=204

1 Solution

drrushi_splunk
Splunk Employee
Splunk Employee

First check the peer's splunkd.log for any messages during the same time as the search-head's DistributedBundleReplicationManager error.

If you do find in the peer's splunkd.log messages such as:

ERROR DistBundleRestHandler - File users/xxx/yyy/local/props.conf in knowledge bundle is either not in white list or else excluded by black list. Bundle /opt/splunk/var/run/searchpeers/ will be removed

...then this means that there must be on the peer a rouge 'distsearch.conf' which does't not explicitly whitelist or blacklist any bundle files ... as a result by default the peer simply rejects the bundle.

To workaround this please remove any distsearch.conf (from system/local OR etc/apps/appname/local) on the peers and restart Splunk.

In version 6.1 a new functionality was added to the peer which allows peers to blacklist/whitelist bundle contents based on locally defined rules (via local distsearch.conf).

View solution in original post

drrushi_splunk
Splunk Employee
Splunk Employee

First check the peer's splunkd.log for any messages during the same time as the search-head's DistributedBundleReplicationManager error.

If you do find in the peer's splunkd.log messages such as:

ERROR DistBundleRestHandler - File users/xxx/yyy/local/props.conf in knowledge bundle is either not in white list or else excluded by black list. Bundle /opt/splunk/var/run/searchpeers/ will be removed

...then this means that there must be on the peer a rouge 'distsearch.conf' which does't not explicitly whitelist or blacklist any bundle files ... as a result by default the peer simply rejects the bundle.

To workaround this please remove any distsearch.conf (from system/local OR etc/apps/appname/local) on the peers and restart Splunk.

In version 6.1 a new functionality was added to the peer which allows peers to blacklist/whitelist bundle contents based on locally defined rules (via local distsearch.conf).

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...