Archive

-0500 ERROR TcpOutputFd - Read error. Connection reset by peer. not not able to forward data into index

rhirasin
Engager

$ tail -f splunkd.log
06-19-2017 06:08:12.823 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer
06-19-2017 06:08:16.540 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.207.140.100_8089_apsrs3355.uhc.com_apsrs3355_128161B5-DBCF-49EE-91B7-406544EB0CDA
06-19-2017 06:08:42.692 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer
06-19-2017 06:09:12.560 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer
06-19-2017 06:09:16.562 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.207.140.100_8089_apsrs3355.uhc.com_apsrs3355_128161B5-DBCF-49EE-91B7-406544EB0CDA
06-19-2017 06:09:42.437 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer
06-19-2017 06:09:49.310 -0500 WARN TcpOutputProc - Forwarding to indexer group indexers blocked for 3500 seconds.
06-19-2017 06:10:12.308 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer
06-19-2017 06:10:16.583 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.207.140.100_8089_apsrs3355.uhc.com_apsrs3355_128161B5-DBCF-49EE-91B7-406544EB0CDA
06-19-2017 06:10:42.177 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer
06-19-2017 06:11:12.050 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer
06-19-2017 06:11:16.606 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.207.140.100_8089_apsrs3355.uhc.com_apsrs3355_128161B5-DBCF-49EE-91B7-406544EB0CDA
06-19-2017 06:11:29.326 -0500 WARN TcpOutputProc - Forwarding to indexer group indexers blocked for 3600 seconds.
06-19-2017 06:11:41.924 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer

Tags (1)
0 Karma

rhirasin
Engager

my issue go resolved. I have to use certs file in my secure env.

thanks for your quick help

0 Karma

arunkchow
New Member

Our org has a small splunk setup. I am trying to secure the splunk with letsencrypt. I have the certs already and put them in /opt/splunk/etc/auth/certs path.
Lets encrypt issues the files as cert.pem, chain.pem, fullchain.pem and privkey.pem.
I pointed to the location of certs in both web.conf and server.conf under /opt/splunk/etc/system/local/ on indexer server and outputs.conf on forwarders.
But I am still getting the same error and forwarders don't forward any data.

0 Karma

jkat54
SplunkTrust
SplunkTrust

I converted your comment to answer. Please mark it as the answer.

0 Karma

jkat54
SplunkTrust
SplunkTrust

It looks like you're trying to send data to the indexer on port 8089. data input is usually on port 9997. Please make sure your specifying the correct forwarding port in your forwarder's outputs.conf.

rhirasin
Engager

here is my output file. its look good. but still its not working

cat outputs.conf

[tcpout]

whitelist all event forwarding

forwardedindex.0.whitelist = .*

null out the following splunk defaults

forwardedindex.1.blacklist =
forwardedindex.2.whitelist =

defaultGroup = indexers

[tcpout:indexers]

Add all indexers on the following line. Separate indexers with a comma, if more than one exists.

server = server1.uhc.com:9997,server2.uhc.com:9997

Test Test

server = apsrd7043:9997

0 Karma

jkat54
SplunkTrust
SplunkTrust

run this command on the forwarder and post the details please.

./splunk btool outputs list --debug

0 Karma

jkat54
SplunkTrust
SplunkTrust

This is the indexer giving the error in the log:

apsrs3355

Yet you have apsrd7043 in the outputs you posted. So the btool command will show us which outputs.conf is pointing to apsrs3355, and then we can assist you on how to solve it... might be as simple as removing the other outputs.conf that points to apsrs3355.

0 Karma