Getting Data In

Problem replicating config (bundle) to search peer ' xx.x.xx.xx:8089 ', Unknown write error.

New Member

i have a relative simple setup. One instance is an indexer, another is search head and heavy forwarder. All seems fine, except when i added the indexer as a search peer. That's when I see the message saying "Problem replicating config (bundle) to search peer ' xx.x.xx.xx:8089 ', Unknown write error." on the search head/heavy forwarder. But no message on the indexers WebUI. Here are the splunkd.log I pulled from both boxes.

Error found on the search head/heavy forwarder

08-12-2019 23:01:29.617 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=HardwareDuplicateMacsToEliminate.csv will attempt to use implicit filename.
08-12-2019 23:01:29.619 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename.
08-12-2019 23:01:29.619 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename.
08-12-2019 23:01:29.629 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware
DuplicateMacsToEliminate.csv will attempt to use implicit filename.
08-12-2019 23:01:29.630 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename.
08-12-2019 23:01:29.631 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename.
08-12-2019 23:01:30.525 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=HardwareDuplicateMacsToEliminate.csv will attempt to use implicit filename.
08-12-2019 23:01:30.526 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename.
08-12-2019 23:01:30.526 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename.
08-12-2019 23:01:38.729 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware
DuplicateMacsToEliminate.csv will attempt to use implicit filename.
08-12-2019 23:01:38.731 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename.
08-12-2019 23:01:38.731 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename.
08-12-2019 23:01:54.815 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=HardwareDuplicateMacsToEliminate.csv will attempt to use implicit filename.
08-12-2019 23:01:54.816 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename.
08-12-2019 23:01:54.817 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename.
08-12-2019 23:01:56.285 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware
DuplicateMacsToEliminate.csv will attempt to use implicit filename.
08-12-2019 23:01:56.286 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename.
08-12-2019 23:01:56.287 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename.
08-12-2019 23:03:41.074 -0400 WARN SSLCommon - Received fatal SSL3 alert. sslstate='SSLv3 read client key exchange A', alertdescription='certificate unknown'.
08-12-2019 23:03:41.075 -0400 WARN HttpListener - Socket error from [My Laptop]:50062 while idling: error:14094416:SSL routines:ssl3readbytes:sslv3 alert certificate unknown
08-12-2019 23:03:41.075 -0400 WARN SSLCommon - Received fatal SSL3 alert. sslstate='SSLv3 read client key exchange A', alertdescription='certificate unknown'.
08-12-2019 23:03:41.075 -0400 WARN HttpListener - Socket error from [My Laptop]:50061 while idling: error:14094416:SSL routines:ssl3readbytes:sslv3 alert certificate unknown
08-12-2019 23:05:09.870 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=HardwareDuplicateMacsToEliminate.csv will attempt to use implicit filename.
08-12-2019 23:05:09.872 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename.
08-12-2019 23:05:09.872 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename.
08-12-2019 23:05:31.286 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware
DuplicateMacsToEliminate.csv will attempt to use implicit filename.
08-12-2019 23:05:31.288 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename.
08-12-2019 23:05:31.288 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename.
08-12-2019 23:13:37.607 -0400 INFO KeyManagerLocalhost - Sending public key to search peer: https://[INDEXER IP]:8089
08-12-2019 23:13:37.612 -0400 INFO KeyManagerLocalhost - Sent public key to search peer: https://[INDEXER IP]:8089
08-12-2019 23:13:37.619 -0400 INFO ServerConfig - Using REMOTESERVERNAME=[SH/HF Host Machine]
08-12-2019 23:13:37.623 -0400 INFO ServerRoles - Declared role=searchhead.
08-12-2019 23:13:43.523 -0400 INFO NetUtils - SSL
write failed. Connection reset by peer
08-12-2019 23:13:43.523 -0400 ERROR DistributedBundleReplicationManager - Unexpected problem while uploading bundle: Unknown write error
08-12-2019 23:13:43.523 -0400 ERROR DistributedBundleReplicationManager - Problem replicating config (bundle) to search peer ' [INDEXER IP]:8089 ', Unknown write error.
08-12-2019 23:13:43.523 -0400 ERROR DistributedBundleReplicationManager - Unable to upload bundle to peer named [INDEXER Host Machine] with uri=https://[INDEXER IP]:8089.
08-12-2019 23:17:00.996 -0400 WARN DistributedPeerManager - Unable to distribute to peer named [INDEXER Host Machine] at uri https://[INDEXER IP]:8089 because replication was unsuccessful. ReplicationStatus: Failed - Failure info: failedbecauseBUNDLEDATATRANSMITFAILURE. Verify connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available. See the Troubleshooting Manual for more information.
08-12-2019 23:17:01.291 -0400 WARN DistributedPeerManager - Unable to distribute to peer named [INDEXER Host Machine] at uri https://[INDEXER IP]:8089 because replication was unsuccessful. ReplicationStatus: Failed - Failure info: failed
becauseBUNDLEDATATRANSMITFAILURE. Verify connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available. See the Troubleshooting Manual for more information.
08-12-2019 23:17:05.469 -0400 INFO NetUtils - SSLwrite failed. Connection reset by peer
08-12-2019 23:17:05.470 -0400 ERROR DistributedBundleReplicationManager - Unexpected problem while uploading bundle: Unknown write error
08-12-2019 23:17:05.470 -0400 ERROR DistributedBundleReplicationManager - Problem replicating config (bundle) to search peer ' [INDEXER IP]:8089 ', Unknown write error.
08-12-2019 23:17:05.470 -0400 ERROR DistributedBundleReplicationManager - Unable to upload bundle to peer named [INDEXER Host Machine] with uri=https://[INDEXER IP]:8089.
08-12-2019 23:18:09.673 -0400 INFO NetUtils - SSL
write failed. Connection reset by peer
08-12-2019 23:18:09.673 -0400 ERROR DistributedBundleReplicationManager - Unexpected problem while uploading bundle: Unknown write error
08-12-2019 23:18:09.673 -0400 ERROR DistributedBundleReplicationManager - Problem replicating config (bundle) to search peer ' [INDEXER IP]:8089 ', Unknown write error.
08-12-2019 23:18:09.673 -0400 ERROR DistributedBundleReplicationManager - Unable to upload bundle to peer named [INDEXER Host Machine] with uri=https://[INDEXER IP]:8089.
08-12-2019 23:21:14.523 -0400 WARN DistributedPeerManager - Unable to distribute to peer named [INDEXER Host Machine] at uri https://[INDEXER IP]:8089 because replication was unsuccessful. ReplicationStatus: Failed - Failure info: failedbecauseBUNDLEDATATRANSMITFAILURE. Verify connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available. See the Troubleshooting Manual for more information.
08-12-2019 23:21:14.526 -0400 WARN DistributedPeerManager - Unable to distribute to peer named [INDEXER Host Machine] at uri https://[INDEXER IP]:8089 because replication was unsuccessful. ReplicationStatus: Failed - Failure info: failed
becauseBUNDLEDATATRANSMITFAILURE. Verify connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available. See the Troubleshooting Manual for more information.
08-12-2019 23:21:18.741 -0400 INFO NetUtils - SSLwrite failed. Connection reset by peer
08-12-2019 23:21:18.741 -0400 ERROR DistributedBundleReplicationManager - Unexpected problem while uploading bundle: Unknown write error
08-12-2019 23:21:18.741 -0400 ERROR DistributedBundleReplicationManager - Problem replicating config (bundle) to search peer ' [INDEXER IP]:8089 ', Unknown write error.
08-12-2019 23:21:18.741 -0400 ERROR DistributedBundleReplicationManager - Unable to upload bundle to peer named [INDEXER Host Machine] with uri=https://[INDEXER IP]:8089.
08-12-2019 23:22:22.986 -0400 INFO NetUtils - SSL
write failed. Connection reset by peer
08-12-2019 23:22:22.986 -0400 ERROR DistributedBundleReplicationManager - Unexpected problem while uploading bundle: Unknown write error
08-12-2019 23:22:22.986 -0400 ERROR DistributedBundleReplicationManager - Problem replicating config (bundle) to search peer ' [INDEXER IP]:8089 ', Unknown write error.
08-12-2019 23:22:22.987 -0400 ERROR DistributedBundleReplicationManager - Unable to upload bundle to peer named [INDEXER Host Machine] with uri=https://[INDEXER IP]:8089.

Error found on the indexer

08-12-2019 23:00:09.326 -0400 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/log/introspection/resourceusage.log'.
08-12-2019 23:13:29.243 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl
state='SSLv3 read client key exchange A', alertdescription='certificate unknown'.
08-12-2019 23:13:29.243 -0400 WARN HttpListener - Socket error from [My Laptop]:50080 while idling: error:14094416:SSL routines:ssl3
readbytes:sslv3 alert certificate unknown
08-12-2019 23:13:29.243 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl
state='SSLv3 read client key exchange A', alertdescription='certificate unknown'.
08-12-2019 23:13:29.243 -0400 WARN HttpListener - Socket error from [My Laptop]:50081 while idling: error:14094416:SSL routines:ssl3
readbytes:sslv3 alert certificate unknown
08-12-2019 23:13:29.731 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl
state='SSLv3 read client key exchange A', alertdescription='certificate unknown'.
08-12-2019 23:13:29.731 -0400 WARN HttpListener - Socket error from [My Laptop]:50083 while idling: error:14094416:SSL routines:ssl3
readbytes:sslv3 alert certificate unknown
08-12-2019 23:13:29.732 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl
state='SSLv3 read client key exchange A', alertdescription='certificate unknown'.
08-12-2019 23:13:29.732 -0400 WARN HttpListener - Socket error from [My Laptop]:50082 while idling: error:14094416:SSL routines:ssl3
readbytes:sslv3 alert certificate unknown
08-12-2019 23:13:32.469 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl
state='SSLv3 read client key exchange A', alertdescription='certificate unknown'.
08-12-2019 23:13:32.469 -0400 WARN HttpListener - Socket error from [My Laptop]:50084 while idling: error:14094416:SSL routines:ssl3
readbytes:sslv3 alert certificate unknown
08-12-2019 23:13:32.475 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl
state='SSLv3 read client key exchange A', alertdescription='certificate unknown'.
08-12-2019 23:13:32.475 -0400 WARN HttpListener - Socket error from [My Laptop]:50085 while idling: error:14094416:SSL routines:ssl3
read_bytes:sslv3 alert certificate unknown
08-12-2019 23:13:37.610 -0400 INFO KeyManagerSearchPeers - Updating public key for search peer: [SH/HF Host Machine]
08-12-2019 23:13:37.611 -0400 INFO KeyManagerSearchPeers - Reading public key for peer: /opt/splunk/etc/auth/distServerKeys/[SH/HF Host Machine]/trusted.pem
08-12-2019 23:13:37.611 -0400 INFO KeyManagerSearchPeers - Finished reading public key for peer: /opt/splunk/etc/auth/distServerKeys/[SH/HF Host Machine]/trusted.pem
08-12-2019 23:13:37.611 -0400 INFO KeyManagerSearchPeers - Finished updating public key for search peer: [SH/HF Host Machine]
08-12-2019 23:13:43.530 -0400 ERROR HttpListener - Exception while processing request from [SH/HF IP]:43164 for /services/receivers/bundle/[SH/HF Host Machine]: Connection closed by peer
08-12-2019 23:13:43.530 -0400 WARN HttpListener - Socket error from [SH/HF IP]:43164 while accessing /services/receivers/bundle/[SH/HF Host Machine]: Broken pipe
08-12-2019 23:14:12.583 -0400 WARN DistributedMetrics - Invalid bundle status
08-12-2019 23:17:05.469 -0400 ERROR HttpListener - Exception while processing request from [SH/HF IP]:43266 for /services/receivers/bundle/[SH/HF Host Machine]: Connection closed by peer
08-12-2019 23:17:05.470 -0400 WARN HttpListener - Socket error from [SH/HF IP]:43266 while accessing /services/receivers/bundle/[SH/HF Host Machine]: Broken pipe
08-12-2019 23:17:18.584 -0400 WARN DistributedMetrics - Invalid bundle status
08-12-2019 23:18:09.678 -0400 ERROR HttpListener - Exception while processing request from [SH/HF IP]:43280 for /services/receivers/bundle/[SH/HF Host Machine]: Connection closed by peer
08-12-2019 23:18:09.678 -0400 WARN HttpListener - Socket error from [SH/HF IP]:43280 while accessing /services/receivers/bundle/[SH/HF Host Machine]: Broken pipe
08-12-2019 23:18:20.584 -0400 WARN DistributedMetrics - Invalid bundle status
08-12-2019 23:21:18.749 -0400 ERROR HttpListener - Exception while processing request from [SH/HF IP]:43324 for /services/receivers/bundle/[SH/HF Host Machine]: Connection closed by peer
08-12-2019 23:21:18.749 -0400 WARN HttpListener - Socket error from [SH/HF IP]:43324 while accessing /services/receivers/bundle/[SH/HF Host Machine]: Broken pipe
08-12-2019 23:21:26.584 -0400 WARN DistributedMetrics - Invalid bundle status

Any Help is much appreciated.

0 Karma