Why am I getting TcpOutputFd error in splunkd.log after heavy forwarder setup?

[vikasshinde]

We have setup a heavy forwarder (for VMware app as a dc node) but we are getting following errors in splunkd.log. Instance is starting correctly.

-0700 ERROR TcpOutputFd - Read error. Connection reset by peer

Any feedback? Thanks.

[horsefez]

Any update on this?
I have a similar problem...

[diogofgm]

can you post you outputs.conf? its probably an issue there

[horsefez]

I try to secure master <-> indexer communication with server certificates signed by our own company rootCA.
Reason is: Forwarding Master-Data to indexers so master does not index any data itself.

I created certificates for my servers according to the splunk documentation.
For the sake of this example I will call the server certificate: servercert.pem
And I will call the rootCA certificate: rootcacert.pem

Lets start at the formatting of the certificates:
the servercert.pem looks like
- servercert in pem format
- privatekey in rsa format (encrypted with secret-key)
- subCAcert in pem format (yes, we have a subCA)
- rootCAcert in pem format

the rootcacert.pem looks like

- rootCAcert in pem format (no subCAcert, only the rootCAcert)

On the master the outputs.conf looks like this

[tcpout]
defaultGroup = Splunk_Indexers

[tcpout:Splunk_Indexer]
server = indexer1:9997,indexer2:9997

[tcpout-server://indexer1.ex.amp.le.de:9997]
sslRootCAPath = /opt/splunk/etc/auth/splunkforwarder/rootcacert.pem
sslCertPath = /opt/splunk/etc/auth/splunkforwarder/servercert.pem
sslPassword = <secret-key>
sslVerifyServerCert = true
sslCommonNameToCheck = indexer1.ex.amp.le.de

[tcpout-server://indexer2.ex.amp.le.de:9997]
sslRootCAPath = /opt/splunk/etc/auth/splunkforwarder/rootcacert.pem
sslCertPath = /opt/splunk/etc/auth/splunkforwarder/servercert.pem
sslPassword = <secret-key>
sslVerifyServerCert = true
sslCommonNameToCheck = indexer2.ex.amp.le.de

On the indexers the inputs.conf (distributed to them over the cluster-bundle) looks like this

[SSL]
rootCA = /opt/splunk/etc/auth/receiver/rootcacert.pem
serverCert = /opt/splunk/etc/auth/receiver/servercert.pem
password = <secret-key>

[splunktcp-ssl:9997]
compressed = true

---

Site note: I created a directory called splunkforwarder and receiver for reason of understandig which certificate resides where on the system.

If I open splunkd.log on the master I find the following error:
ERROR TcpOutputFd - Read error. Connection reset by peer

On the indexers splunkd.log looks like this:
ERROR TcpInputProc - Error encountered for connection from src=:38953. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

Help would be hugely apprechiated, because I work on this problem since monday!

[vikasshinde]

inputs.conf:

[default]
host = xyz [splunktcp-ssl://9997]
disabled=0

outputs.conf: [tcpout]
defaultGroup = indexers
indexAndForward = 1 [tcpout:indexers]
disabled = false
autoLB = true
server = indexer1:9997, indexer2:9997, indexer3:9997

[tcpout-server://indexer1:9997]
disabled = 0
[tcpout-server:/indexer2:9997]
disabled = 0
[tcpout-server://indexer3:9997]
disabled = 0

[rsennett_splunk]

You're going to want to post inputs.conf and outputs.conf here... The error is most likely caused by a configuration error (usually an accidental loop).