i have a relative simple setup. One instance is an indexer, another is search head and heavy forwarder. All seems fine, except when i added the indexer as a search peer. That's when I see the message saying "Problem replicating config (bundle) to search peer ' xx.x.xx.xx:8089 ', Unknown write error." on the search head/heavy forwarder. But no message on the indexers WebUI. Here are the splunkd.log I pulled from both boxes. Error found on the search head/heavy forwarder 08-12-2019 23:01:29.617 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware_DuplicateMacsToEliminate.csv will attempt to use implicit filename. 08-12-2019 23:01:29.619 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename. 08-12-2019 23:01:29.619 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename. 08-12-2019 23:01:29.629 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware_DuplicateMacsToEliminate.csv will attempt to use implicit filename. 08-12-2019 23:01:29.630 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename. 08-12-2019 23:01:29.631 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename. 08-12-2019 23:01:30.525 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware_DuplicateMacsToEliminate.csv will attempt to use implicit filename. 08-12-2019 23:01:30.526 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename. 08-12-2019 23:01:30.526 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename. 08-12-2019 23:01:38.729 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware_DuplicateMacsToEliminate.csv will attempt to use implicit filename. 08-12-2019 23:01:38.731 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename. 08-12-2019 23:01:38.731 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename. 08-12-2019 23:01:54.815 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware_DuplicateMacsToEliminate.csv will attempt to use implicit filename. 08-12-2019 23:01:54.816 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename. 08-12-2019 23:01:54.817 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename. 08-12-2019 23:01:56.285 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware_DuplicateMacsToEliminate.csv will attempt to use implicit filename. 08-12-2019 23:01:56.286 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename. 08-12-2019 23:01:56.287 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename. 08-12-2019 23:03:41.074 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'. 08-12-2019 23:03:41.075 -0400 WARN HttpListener - Socket error from [My Laptop]:50062 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown 08-12-2019 23:03:41.075 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'. 08-12-2019 23:03:41.075 -0400 WARN HttpListener - Socket error from [My Laptop]:50061 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown 08-12-2019 23:05:09.870 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware_DuplicateMacsToEliminate.csv will attempt to use implicit filename. 08-12-2019 23:05:09.872 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename. 08-12-2019 23:05:09.872 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename. 08-12-2019 23:05:31.286 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware_DuplicateMacsToEliminate.csv will attempt to use implicit filename. 08-12-2019 23:05:31.288 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename. 08-12-2019 23:05:31.288 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename. 08-12-2019 23:13:37.607 -0400 INFO KeyManagerLocalhost - Sending public key to search peer: https://[INDEXER IP]:8089 08-12-2019 23:13:37.612 -0400 INFO KeyManagerLocalhost - Sent public key to search peer: https://[INDEXER IP]:8089 08-12-2019 23:13:37.619 -0400 INFO ServerConfig - Using REMOTE_SERVER_NAME=[SH/HF Host Machine] 08-12-2019 23:13:37.623 -0400 INFO ServerRoles - Declared role=search_head. 08-12-2019 23:13:43.523 -0400 INFO NetUtils - SSL_write failed. Connection reset by peer 08-12-2019 23:13:43.523 -0400 ERROR DistributedBundleReplicationManager - Unexpected problem while uploading bundle: Unknown write error 08-12-2019 23:13:43.523 -0400 ERROR DistributedBundleReplicationManager - Problem replicating config (bundle) to search peer ' [INDEXER IP]:8089 ', Unknown write error. 08-12-2019 23:13:43.523 -0400 ERROR DistributedBundleReplicationManager - Unable to upload bundle to peer named [INDEXER Host Machine] with uri=https://[INDEXER IP]:8089. 08-12-2019 23:17:00.996 -0400 WARN DistributedPeerManager - Unable to distribute to peer named [INDEXER Host Machine] at uri https://[INDEXER IP]:8089 because replication was unsuccessful. ReplicationStatus: Failed - Failure info: failed_because_BUNDLE_DATA_TRANSMIT_FAILURE. Verify connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available. See the Troubleshooting Manual for more information. 08-12-2019 23:17:01.291 -0400 WARN DistributedPeerManager - Unable to distribute to peer named [INDEXER Host Machine] at uri https://[INDEXER IP]:8089 because replication was unsuccessful. ReplicationStatus: Failed - Failure info: failed_because_BUNDLE_DATA_TRANSMIT_FAILURE. Verify connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available. See the Troubleshooting Manual for more information. 08-12-2019 23:17:05.469 -0400 INFO NetUtils - SSL_write failed. Connection reset by peer 08-12-2019 23:17:05.470 -0400 ERROR DistributedBundleReplicationManager - Unexpected problem while uploading bundle: Unknown write error 08-12-2019 23:17:05.470 -0400 ERROR DistributedBundleReplicationManager - Problem replicating config (bundle) to search peer ' [INDEXER IP]:8089 ', Unknown write error. 08-12-2019 23:17:05.470 -0400 ERROR DistributedBundleReplicationManager - Unable to upload bundle to peer named [INDEXER Host Machine] with uri=https://[INDEXER IP]:8089. 08-12-2019 23:18:09.673 -0400 INFO NetUtils - SSL_write failed. Connection reset by peer 08-12-2019 23:18:09.673 -0400 ERROR DistributedBundleReplicationManager - Unexpected problem while uploading bundle: Unknown write error 08-12-2019 23:18:09.673 -0400 ERROR DistributedBundleReplicationManager - Problem replicating config (bundle) to search peer ' [INDEXER IP]:8089 ', Unknown write error. 08-12-2019 23:18:09.673 -0400 ERROR DistributedBundleReplicationManager - Unable to upload bundle to peer named [INDEXER Host Machine] with uri=https://[INDEXER IP]:8089. 08-12-2019 23:21:14.523 -0400 WARN DistributedPeerManager - Unable to distribute to peer named [INDEXER Host Machine] at uri https://[INDEXER IP]:8089 because replication was unsuccessful. ReplicationStatus: Failed - Failure info: failed_because_BUNDLE_DATA_TRANSMIT_FAILURE. Verify connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available. See the Troubleshooting Manual for more information. 08-12-2019 23:21:14.526 -0400 WARN DistributedPeerManager - Unable to distribute to peer named [INDEXER Host Machine] at uri https://[INDEXER IP]:8089 because replication was unsuccessful. ReplicationStatus: Failed - Failure info: failed_because_BUNDLE_DATA_TRANSMIT_FAILURE. Verify connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available. See the Troubleshooting Manual for more information. 08-12-2019 23:21:18.741 -0400 INFO NetUtils - SSL_write failed. Connection reset by peer 08-12-2019 23:21:18.741 -0400 ERROR DistributedBundleReplicationManager - Unexpected problem while uploading bundle: Unknown write error 08-12-2019 23:21:18.741 -0400 ERROR DistributedBundleReplicationManager - Problem replicating config (bundle) to search peer ' [INDEXER IP]:8089 ', Unknown write error. 08-12-2019 23:21:18.741 -0400 ERROR DistributedBundleReplicationManager - Unable to upload bundle to peer named [INDEXER Host Machine] with uri=https://[INDEXER IP]:8089. 08-12-2019 23:22:22.986 -0400 INFO NetUtils - SSL_write failed. Connection reset by peer 08-12-2019 23:22:22.986 -0400 ERROR DistributedBundleReplicationManager - Unexpected problem while uploading bundle: Unknown write error 08-12-2019 23:22:22.986 -0400 ERROR DistributedBundleReplicationManager - Problem replicating config (bundle) to search peer ' [INDEXER IP]:8089 ', Unknown write error. 08-12-2019 23:22:22.987 -0400 ERROR DistributedBundleReplicationManager - Unable to upload bundle to peer named [INDEXER Host Machine] with uri=https://[INDEXER IP]:8089. Error found on the indexer 08-12-2019 23:00:09.326 -0400 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/log/introspection/resource_usage.log'. 08-12-2019 23:13:29.243 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'. 08-12-2019 23:13:29.243 -0400 WARN HttpListener - Socket error from [My Laptop]:50080 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown 08-12-2019 23:13:29.243 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'. 08-12-2019 23:13:29.243 -0400 WARN HttpListener - Socket error from [My Laptop]:50081 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown 08-12-2019 23:13:29.731 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'. 08-12-2019 23:13:29.731 -0400 WARN HttpListener - Socket error from [My Laptop]:50083 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown 08-12-2019 23:13:29.732 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'. 08-12-2019 23:13:29.732 -0400 WARN HttpListener - Socket error from [My Laptop]:50082 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown 08-12-2019 23:13:32.469 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'. 08-12-2019 23:13:32.469 -0400 WARN HttpListener - Socket error from [My Laptop]:50084 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown 08-12-2019 23:13:32.475 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'. 08-12-2019 23:13:32.475 -0400 WARN HttpListener - Socket error from [My Laptop]:50085 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown 08-12-2019 23:13:37.610 -0400 INFO KeyManagerSearchPeers - Updating public key for search peer: [SH/HF Host Machine] 08-12-2019 23:13:37.611 -0400 INFO KeyManagerSearchPeers - Reading public key for peer: /opt/splunk/etc/auth/distServerKeys/[SH/HF Host Machine]/trusted.pem 08-12-2019 23:13:37.611 -0400 INFO KeyManagerSearchPeers - Finished reading public key for peer: /opt/splunk/etc/auth/distServerKeys/[SH/HF Host Machine]/trusted.pem 08-12-2019 23:13:37.611 -0400 INFO KeyManagerSearchPeers - Finished updating public key for search peer: [SH/HF Host Machine] 08-12-2019 23:13:43.530 -0400 ERROR HttpListener - Exception while processing request from [SH/HF IP]:43164 for /services/receivers/bundle/[SH/HF Host Machine]: Connection closed by peer 08-12-2019 23:13:43.530 -0400 WARN HttpListener - Socket error from [SH/HF IP]:43164 while accessing /services/receivers/bundle/[SH/HF Host Machine]: Broken pipe 08-12-2019 23:14:12.583 -0400 WARN DistributedMetrics - Invalid bundle status 08-12-2019 23:17:05.469 -0400 ERROR HttpListener - Exception while processing request from [SH/HF IP]:43266 for /services/receivers/bundle/[SH/HF Host Machine]: Connection closed by peer 08-12-2019 23:17:05.470 -0400 WARN HttpListener - Socket error from [SH/HF IP]:43266 while accessing /services/receivers/bundle/[SH/HF Host Machine]: Broken pipe 08-12-2019 23:17:18.584 -0400 WARN DistributedMetrics - Invalid bundle status 08-12-2019 23:18:09.678 -0400 ERROR HttpListener - Exception while processing request from [SH/HF IP]:43280 for /services/receivers/bundle/[SH/HF Host Machine]: Connection closed by peer 08-12-2019 23:18:09.678 -0400 WARN HttpListener - Socket error from [SH/HF IP]:43280 while accessing /services/receivers/bundle/[SH/HF Host Machine]: Broken pipe 08-12-2019 23:18:20.584 -0400 WARN DistributedMetrics - Invalid bundle status 08-12-2019 23:21:18.749 -0400 ERROR HttpListener - Exception while processing request from [SH/HF IP]:43324 for /services/receivers/bundle/[SH/HF Host Machine]: Connection closed by peer 08-12-2019 23:21:18.749 -0400 WARN HttpListener - Socket error from [SH/HF IP]:43324 while accessing /services/receivers/bundle/[SH/HF Host Machine]: Broken pipe 08-12-2019 23:21:26.584 -0400 WARN DistributedMetrics - Invalid bundle status Any Help is much appreciated. ... View more

I have a new data set being pushed via Splunk REST API. Because I am using selective indexing, I have some indexes stored locally, some don't. It's all set via _INDEX_AND_FORWARD_ROUTING . However, it doesn't seem to work for this new data set coming in from the REST API. Others are not via the REST API, and other data show up during the search. Here is what I put in as a new stanza in inputs.conf. [monitor:///opt/splunk/var/lib/splunk/new-ind/db] index = new-ind source = restAPI sourcetype = web_event _INDEX_AND_FORWARD_ROUTING = new-ind Any help will be appreciated. ... View more