topic Re: Why am I receiving an error when deploying a new Splunk forwarder? in Getting Data In

Why am I receiving an error when deploying a new Splunk forwarder?

gchotlineinfo — Fri, 13 Apr 2018 14:36:15 GMT

Hi,

I try to deploy a new forwarder since i've updated my indexer to 7.0.3. I got some problems and i found my answers on this forum.
But I haven't been able to solve, below the error message in the splunkd.log

04-13-2018 13:22:44.069 +0000 INFO  TcpOutputProc - Removing quarantine from idx=IPAddress:9997
04-13-2018 13:22:44.072 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
04-13-2018 13:22:44.074 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
04-13-2018 13:22:44.074 +0000 WARN  TcpOutputProc - Applying quarantine to ip=IPAddress port=9997 _numberOfFailures=2
04-13-2018 13:22:51.491 +0000 INFO  HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IPAddress_8089_Hostname_ShortName_E4BC416F-983F-4CEF-AA47-45BA28ED0FF3
04-13-2018 13:22:51.503 +0000 INFO  HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IPAddress_8089_Hostname_ShortName_E4BC416F-983F-4CEF-AA47-45BA28ED0FF3
04-13-2018 13:23:51.505 +0000 INFO  HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IPAddress_8089_Hostname_ShortName_E4BC416F-983F-4CEF-AA47-45BA28ED0FF3
04-13-2018 13:23:51.517 +0000 INFO  HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IPAddress_8089_Hostname_ShortName_E4BC416F-983F-4CEF-AA47-45BA28ED0FF3
04-13-2018 13:24:17.921 +0000 WARN  TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group splunkssl has been blocked for 600 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

And on my indexer :

04-13-2018 15:24:50.665 +0200 INFO  ClientSessionsManager:Listener_AppEvents - Received count=1 AppEvent from DC ip=172.25.225.49 name=E4BC416F-983F-4CEF-AA47-45BA28ED0FF3
04-13-2018 15:26:42.372 +0200 ERROR TcpInputProc - Error encountered for connection from src=IPAddress:47781. error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

Port 8089, 9997 listen and telnet in between works.
Forwarder outputs.conf

[tcpout]

[tcpout:splunkssl]
server = indexer:9997

[tcpout-server://indexer:9997]
sslCertPath = /opt/splunkforwarder/etc/certs/splunk-sys-forwarder.pem
sslCommonNameToCheck = indexer
sslPassword = CaCertPassword
sslRootCAPath = /opt/splunkforwarder/etc/certs/cacert.pem
sslVerifyServerCert = false

Indexer inputs.conf

[splunktcp-ssl:9997]
disabled = 0
connection_host = ip

[SSL]

serverCert = /opt/splunk/etc/certs/splunk-sys-indexer.pem
sslPassword = CaCertPassword
requireClientCert = false

Re: Why am I receiving an error when deploying a new Splunk forwarder?

s2_splunk — Fri, 13 Apr 2018 19:59:03 GMT

Error encountered for connection from src=IPAddress:47781. error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

Indicates that your forwarder is trying to use an SSL version not supported by your indexer. What version did you upgrade from on your indexer and what version is your forwarder?
As of 6.6 we will default to TLS1.2 and if your forwarder requests a lower SSL version you will see this message. Review the docs to see if the workaround works for you; or upgrade your UF to a version post 6.6.

Re: Why am I receiving an error when deploying a new Splunk forwarder?

gchotlineinfo — Mon, 16 Apr 2018 08:46:54 GMT

I upgraded from 6.2.2 to 7.0.3 for indexer and forwarders. I checked with the command : /opt/splunk/bin/splunk cmd btool inputs list --debug

/opt/splunk/etc/system/default/inputs.conf sslVersions = tls1.2

And on forwarder :

/opt/splunkforwarder/etc/system/default/outputs.conf sslVersions = tls1.2

Re: Why am I receiving an error when deploying a new Splunk forwarder?

tlam_splunk — Mon, 16 Apr 2018 09:19:50 GMT

Please check the cipherSuite parameter and see they are matched in Indexer and forwarder

Re: Why am I receiving an error when deploying a new Splunk forwarder?

gchotlineinfo — Mon, 16 Apr 2018 09:26:14 GMT

Indeed, there is a difference :

Indexer

ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256

Forwarder

ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256

I added these parameters on both sides, I have the same result.

sslVersions = tls1.2
cipherSuite = AES256-SHA256:DHE-RSA-AES256-SHA256

Re: Why am I receiving an error when deploying a new Splunk forwarder?

afroz — Mon, 23 Jul 2018 12:03:37 GMT

splunk forwarders version must be equal or lower than indexers. Fix that problem, this error won't come.

Re: Why am I receiving an error when deploying a new Splunk forwarder?

mkolkebeck — Thu, 25 Oct 2018 04:34:28 GMT

Per the link below, it's a best practice to have a higher indexer version, but not required.
http://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/Compatibilitybetweenforwardersandindexers

Re: Why am I receiving an error when deploying a new Splunk forwarder?

mkolkebeck — Thu, 25 Oct 2018 04:39:01 GMT

I'd recommend putting your ssl settings in outputs.conf under your [tcpout:splunkssl]. Per the spec, the [tcpout-server://indexer:9997] stanza is optional, unless you need common name checking of a single instance across a distributed indexer deployment.

It's also possible that you may have an invalid sslPassword or bad certificate.

You should also verify that you can connect via s_client:

./splunk cmd openssl s_client -connect indexer:9997

Re: Why am I receiving an error when deploying a new Splunk forwarder?

bcyates — Mon, 18 Mar 2019 17:08:49 GMT

I downvoted this post because it is not true, per splunk docs