Knowledge bundles not built

[thomasbader]

Having an issue that Splunk doesn't build my knowledge bundles. My setup: One indexer cluster and two standalone search heads (no SH cluster). Both search heads use indexer discovery and the setup used to work fine. Until recently the knowledge bundle of one of the two search heads stopped getting updated on the indexers.

I observe the following:

• All indexers always have an up to date knowledge bundle from the first search head in /opt/splunk/var/run/searchpeers, while the bundle from the second search head no longer gets updated and is outdated.
• When running "splunk show bundle-replication-config" on the two search heads, both show an identical config
• When running "splunk show bundle-replication-status", one search head shows a fully functional replication, while the other search head states "No knowledge bundle replication cycle status is available yet."
• The search head that shows the error with the replication cycle status has no local knowledge bundle in /opt/splunk/var/run/ (while the other search head indeed has it). Therefore I guess that there's not a problem on the channel between search head and indexer, but some interna on the search head is dysfunctional and no longer builds the bundles in the first place.

I did all the usual checks (reboot, filesystem permissions, btool check, ...). On the broken search head, I moved all local apps out of SPLUNK_HOME/etc/apps and emptied SPLUNK_HOME/etc/users and restarted, but the knowledge bundle still wasn't getting build.

In log.cfg on the SH I set DistributedBundleReplicationManager, BundleReplicationProvider, ClassicBundleReplicationProvider, CascadingBundleReplicationProvider, RFSBundleReplicationProvider, RFSManager to DEBUG, but this didn't provide any insights.

Any ideas about where we could search further?