Knowledge bundle replication failed, checksum mismatch

[tsabu]

Hello,

I'm upgrading a search head from 7.3.0 to 8.2.1. First I upgraded it to 8.1.5 and I didn't experienced any problems. Then I upgraded to 8.2.1 and the knowledge bundle replication to the search peers failed with the following errors in the logs.

In search head splunkd.log:
08-23-2021 18:48:56.228 +0200 WARN BundleTransaction [2589 BundleReplThreadPoolWorker-1] - Upload bundle="/opt/splunk/current/var/run/sh01-1629737334.bundle" to peer name=idx01 uri=https://10.10.22.14:8089 failed; http_status=409 http_description="Conflict"
08-23-2021 18:48:56.234 +0200 ERROR ClassicBundleReplicationProvider [2589 BundleReplThreadPoolWorker-1] - Unable to upload bundle to peer named idx01 with uri=https://10.10.22.14:8089.

In indexers splunkd.log:
08-23-2021 18:48:56.225 +0200 ERROR DistBundleRestHandler - Checksum mismatch: received copy of bundle="/opt/splunk/var/run/searchpeers/sh01-1629737334.bundle" has transferred_checksum=15251024310319607191 instead of checksum=5204570444500435281 -- removing temporary file="/opt/splunk/var/run/searchpeers/sh01-1629737334.bundle.c2ead49153e7b186.tmp". This should be fixed with the next knowledge bundle replication. If it persists, please check your filesystem and network interface for errors.

The bundle size is not big, but the size reported in the .info is quite different from the size on the filesystem:

[splunk@sh01 run]$ ls -l
...
-rw------- 1 splunk splunk 4280079 Aug 23 18:48 sh01-1629737334.bundle
-rw------- 1 splunk splunk 42 Aug 23 18:48 sh01-1629737334.bundle.info

[splunk@sh01 run]$ cat sh01-1629737334.bundle.info
checksum,size
5204570444500435281,6574080

The indexers are in a cluster and all nodes are running version 7.3.0. I know Splunk recommends the manager node to be higher or equal version but I'm validating some custom apps on a test search head, which I wanted to do in version 8.2.
In another not production environment a search head on 8.2 works (no bundle replication problems) with indexers 7.3.0.

[manjunathmeti]

Hi @tsabu,

You need to upgrade indexer servers to 8.x. Check this link https://docs.splunk.com/Documentation/Splunk/8.2.2/DistSearch/Distsearchsystemrequirements#Compatibi... 

It clearly mentions A 8.2 search head is not compatible with a 7.3 search peer.

If this reply helps you, a like would be appreciated.