Getting Data In

Getting this Unable to distribute to peer named xxxx at uri https://xxxxxx.com:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_REMOTE_CHKSUM_UNMATCHED: Remote checksum does not match:

Hemnaath
Motivator

Hi Team,

I'm seeing a weird issue in splunk– I am getting the below error on my search head,

Splunk Environment details:
We have 5 individual indexer instance, 5 individual indexer, 3 search head cluster, deployer and deployment servers with splunk 6.6.1 version .

warning message: 
Unable to distribute to peer named xxxx at uri https://xxxxxx.com:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_REMOTE_CHKSUM_UNMATCHED: Remote checksum does not match: remote(xxxx, 6326192254332865118, 1524083405) != local(9229541009365200778, 1524083377) Please verify connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information. 

This pop up message was triggered from one of my indexer instance while trying to do search activity. SO checked the status of the splunk services are running fine. For futher investigation checked the splunkd.log details for ERROR /WARN on the indexer/search head instances and got the below details

Search Query: 
index="_internal" sourcetype=splunkd log_level=WARN | dedup message | table  log_level message 

Events 
04-19-2018 04:10:10.318 -0400 WARN  LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 13561 - data_source="/opt/splunk/var/log/splunk/remote_searches.log", data_host="splunk03", data_sourcetype="splunkd_remote_searches"

Truncating line because limit of 10000 bytes has been exceeded with a line length >= 11525 - data_source="/opt/splunk/var/log/splunk/remote_searches.log", data_host="splunk03", data_sourcetype="splunkd_remote_searches"

Similarly when checked the splunkd.log details from the search head, could get the below WARN details.

index=_internal host="test0*" log_level=WARN | dedup message | table log_level message

WARN    Asynchronous bundle replication to 5 peer(s) succeeded; however it took too long (longer than 10 seconds): elapsed_ms=32193, tar_elapsed_ms=7839 , for 5 peer(s), bundle_replication_mode="delta", bundle_file_name=C090FDA2-105E-4875-A110-3F13FF986151-1524124411-1524124501.delta, bundle_size=330KB, replication_id=1524124501, replication_reason="async replication allowed"
WARN    Asynchronous bundle replication to 5 peer(s) succeeded; however it took too long (longer than 10 seconds): elapsed_ms=30111, tar_elapsed_ms=8025 , for 5 peer(s), bundle_replication_mode="delta", bundle_file_name=C090FDA2-105E-4875-A110-3F13FF986151-1524124320-1524124411.delta, bundle_size=320KB, replication_id=1524124411, replication_reason="async replication allowed"
WARN    Asynchronous bundle replication to 5 peer(s) succeeded; however it took too long (longer than 10 seconds): elapsed_ms=30970, tar_elapsed_ms=9440 , for 5 peer(s), bundle_replication_mode="delta", bundle_file_name=C090FDA2-105E-4875-A110-3F13FF986151-1524124230-1524124320.delta, bundle_size=320KB, replication_id=1524124320, replication_reason="async replication allowed"
WARN    Asynchronous bundle replication to 5 peer(s) succeeded; however it took too long (longer than 10 seconds): elapsed_ms=29926, tar_elapsed_ms=7447 , for 4 peer(s), bundle_replication_mode="delta", bundle_file_name=C090FDA2-105E-4875-A110-3F13FF986151-1524124139-1524124230.delta, bundle_size=320KB, replication_id=1524124230, replication_reason="async replication allowed"

Note: There is no problem in performing the searching activities as we are able to get the required details based on the search string criteria, but not sure why this warning message is popping out and how to fix this issue.

Kindly guide me to fix this issue.

0 Karma
1 Solution

Hemnaath
Motivator

thanks, for sharing the link. Issue got fixed after pushing an another app from the deployer which we were working for another request, when deployed it restarted all the search head and then we did not see the above pop-up again in the search head.

View solution in original post

0 Karma

Hemnaath
Motivator

thanks, for sharing the link. Issue got fixed after pushing an another app from the deployer which we were working for another request, when deployed it restarted all the search head and then we did not see the above pop-up again in the search head.

0 Karma

gdavismn
New Member

I believe this was "supposed" to be fixed already,. but check this thread.

https://answers.splunk.com/answers/550473/replication-was-unsuccessful-failed-because-remote.html

0 Karma

pbrinkman
Path Finder

hi,

The link provided below by gdavismn does not return anything,

https://answers.splunk.com/answers/550473/replication-was-unsuccessful-failed-because-remote.htm

the link is missing the last l on html
add the l and the link works. See full link below

https://answers.splunk.com/answers/550473/replication-was-unsuccessful-failed-because-remote.html

0 Karma
Get Updates on the Splunk Community!

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...