Getting this Unable to distribute to peer named xxxx at uri https://xxxxxx.com:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_REMOTE_CHKSUM_UNMATCHED: Remote checksum does not match:

[Hemnaath]

Hi Team,

I'm seeing a weird issue in splunk– I am getting the below error on my search head,

Splunk Environment details:
We have 5 individual indexer instance, 5 individual indexer, 3 search head cluster, deployer and deployment servers with splunk 6.6.1 version .

warning message: 
Unable to distribute to peer named xxxx at uri https://xxxxxx.com:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_REMOTE_CHKSUM_UNMATCHED: Remote checksum does not match: remote(xxxx, 6326192254332865118, 1524083405) != local(9229541009365200778, 1524083377) Please verify connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information. 

This pop up message was triggered from one of my indexer instance while trying to do search activity. SO checked the status of the splunk services are running fine. For futher investigation checked the splunkd.log details for ERROR /WARN on the indexer/search head instances and got the below details

Search Query: 
index="_internal" sourcetype=splunkd log_level=WARN | dedup message | table  log_level message 

Events 
04-19-2018 04:10:10.318 -0400 WARN  LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 13561 - data_source="/opt/splunk/var/log/splunk/remote_searches.log", data_host="splunk03", data_sourcetype="splunkd_remote_searches"

Truncating line because limit of 10000 bytes has been exceeded with a line length >= 11525 - data_source="/opt/splunk/var/log/splunk/remote_searches.log", data_host="splunk03", data_sourcetype="splunkd_remote_searches"

Similarly when checked the splunkd.log details from the search head, could get the below WARN details.

index=_internal host="test0*" log_level=WARN | dedup message | table log_level message

WARN    Asynchronous bundle replication to 5 peer(s) succeeded; however it took too long (longer than 10 seconds): elapsed_ms=32193, tar_elapsed_ms=7839 , for 5 peer(s), bundle_replication_mode="delta", bundle_file_name=C090FDA2-105E-4875-A110-3F13FF986151-1524124411-1524124501.delta, bundle_size=330KB, replication_id=1524124501, replication_reason="async replication allowed"
WARN    Asynchronous bundle replication to 5 peer(s) succeeded; however it took too long (longer than 10 seconds): elapsed_ms=30111, tar_elapsed_ms=8025 , for 5 peer(s), bundle_replication_mode="delta", bundle_file_name=C090FDA2-105E-4875-A110-3F13FF986151-1524124320-1524124411.delta, bundle_size=320KB, replication_id=1524124411, replication_reason="async replication allowed"
WARN    Asynchronous bundle replication to 5 peer(s) succeeded; however it took too long (longer than 10 seconds): elapsed_ms=30970, tar_elapsed_ms=9440 , for 5 peer(s), bundle_replication_mode="delta", bundle_file_name=C090FDA2-105E-4875-A110-3F13FF986151-1524124230-1524124320.delta, bundle_size=320KB, replication_id=1524124320, replication_reason="async replication allowed"
WARN    Asynchronous bundle replication to 5 peer(s) succeeded; however it took too long (longer than 10 seconds): elapsed_ms=29926, tar_elapsed_ms=7447 , for 4 peer(s), bundle_replication_mode="delta", bundle_file_name=C090FDA2-105E-4875-A110-3F13FF986151-1524124139-1524124230.delta, bundle_size=320KB, replication_id=1524124230, replication_reason="async replication allowed"

Note: There is no problem in performing the searching activities as we are able to get the required details based on the search string criteria, but not sure why this warning message is popping out and how to fix this issue.

Kindly guide me to fix this issue.

[Hemnaath · Accepted solution]

thanks, for sharing the link. Issue got fixed after pushing an another app from the deployer which we were working for another request, when deployed it restarted all the search head and then we did not see the above pop-up again in the search head.

[gdavismn]

I believe this was "supposed" to be fixed already,. but check this thread.

https://answers.splunk.com/answers/550473/replication-was-unsuccessful-failed-because-remote.html

[pbrinkman]

hi,

The link provided below by gdavismn does not return anything,

https://answers.splunk.com/answers/550473/replication-was-unsuccessful-failed-because-remote.htm

the link is missing the last l on html
add the l and the link works. See full link below

https://answers.splunk.com/answers/550473/replication-was-unsuccessful-failed-because-remote.html