How to resolve "SSL23_GET_CLIENT_HELLO:unknown protocol" error on our indexer?

[kermitshort]

I'm setting up a Splunk Indexer (Splunk Enterprise 6.4.1) on CentOS 6.8 64-bit. I do have the Splunk Add-on for Microsoft Windows installed on the indexer.

My (/opt/splunk/etc/system/local/)inputs.conf (owned by root.root, 600 permissions):

[default]
host = splunkidx

[splunktcp-ssl:9997]
disabled = 0
queueSize = 1000

[SSL]
password = <<HIDDEN>>
requireClientCert = false
rootCA = /opt/splunk/etc/auth/ssocerts/FACE-CA-root.pem
serverCert = /opt/splunk/etc/auth/ssocerts/splunkidx-idx.pem
sslVersions = tls, =tls1.0

I'm trying to receive data from Windows 7 64-bit Universal Forwarder.
My (C:\Program Files\SplunkUniversalForwarder\etc\system\local)outputs.conf (default permissions and ownership):

[tcpout]
defaultGroup = default-autolb-group
maxQueueSize = 1000

[tcpout:default-autolb-group]
server = 192.168.58.11:9997
sslCertPath = etc\auth\ssocerts\splunk-forwarder-agent.pem
sslPassword = <<HIDDEN>>
sslRootCAPath = etc\auth\ssocerts\FACE-CA-root.pem
sslVerifyServerCert = true
sslCommonNameToCheck = splunkidx.<<HIDDEN>>

The Splunk Indexer starts fine. I can even enable the web interface and access it.

The Splunk Forwarder starts fine. No errors during startup.

When the Forwarder has finished starting and begins attempting to contact the indexer, I see the following log information.
My (C:\Program Files\SplunkUniversalForwarder\var\log\splunk)splunkd.log:

07-27-2016 07:30:44.060 -0600 ERROR TcpOutputFd - Read error. An established connection was aborted by the software in your host machine.
07-27-2016 07:31:14.074 -0600 ERROR TcpOutputFd - Read error. An existing connection was forcibly closed by the remote host.

This appears in random duplicate stanzas throughout the log until I kill splunkd.

On the Indexer, I see the following corresponding log entries.
My (/opt/splunk/var/log/splunk/)splunkd.log:

07-27-2016 08:30:39.697 -0600 ERROR TcpInputProc - Error encountered for connection from src=192.168.50.151:65251. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
07-27-2016 08:31:09.697 -0600 ERROR TcpInputProc - Error encountered for connection from src=192.168.50.151:65252. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

Since I don't have enough forum points, I can't post links here, but I've verified settings and suggestions posted in other answers on this forum...specifically:
- The server.conf is configured with:

...
[sslConfig]
sslKeysfilePassword = <<HIDDEN>>
useClientSSLCompression = true
...

The version of OpenSSL on the forwarder is: OpenSSL 1.0.2h-fips 3 May 2016
The version of OpenSSL on the indexer is: openssl 1.0.1e

I have appended the server keys to the server cert in a new file, and that's what I'm using in the above configurations.

At this point I'm entirely out of ideas, but I really need to get these Windows systems talking to Splunk Enterprise! Your advice is greatly appreciated!!

[Topherino]

I got the same error when sslPassword was wrong for a particular outputs.conf entry.   I determined even if you disable cert validation sslCertPath, sslPassword, and sslRootCAPath must all point at valid files, and they weren't inherited from a higher level so they must be set if the remote host is ssl.   You can check if they are set elsewhere you can copy and paste into your config by using $SPLUNK_HOME/bin/splunk btool outputs list --debug

[moorvogi]

we have this issue also but we only have one protocol listed. the thing I'll note is, the protocol is wrapped in quotation marks. see example:

sslVersions = "tls1.2"
instead of
sslVersions = tls1.2

the servers are able to talk to each other just fine so i'm having a hard time accepting that the quote marks are causing the problem but i dont see it in any of the examples.

[jimmoriarty]

Coming late to the party, and in response to the query in regards to the Microsoft Active Directory integrated Certificate Authority (and possibly other CA's) - you need to ensure that the resulting certificate is allowed to be used for Server authentication.

First - all the points made by kermitshort are correct - so start there, however...

Having only seen one instance of MS AD Cert Services (well two, one test and one prod, but both setup in a similar way), I can only assume some things are common - mainly that when you submit your csr, you're asked which Certificate Template to use. These appear to be site specific, but you'll need one that returns a cert with the rights to be used for Server Authentication - at least on the receiving (inputs) side of the communication (e.g. the indexers)

The only way I've found out how to confirm this is once I've received the new certificate, I run it through openssl:

splunk cmd openssl x509 -in <newcert> -text -noout

and check the output for the Extended Key Usage for the Server Authentication usage:
X509v3 Extended Key Usage:
TLS Web Client Authentication, TLS Web Server Authentication

Discuss the template capabilities with the owner of your Microsoft PKI setup...

Note: without this, I was getting the following errors...

[From Indexer]
08-17-2017 11:02:10.257 +1000 ERROR TcpInputProc - Error encountered for connection from src=. error:14094413:SSL routines:ssl3_read_bytes:sslv3 alert unsupported certificate

[From universal forwarder]
08-17-2017 10:58:10.742 +1000 ERROR TcpOutputFd - Connection to host= failed. sock_error = 0. SSL Error = error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

[kermitshort]

Reading the manual really worked out. I guess I should have done that first. I can now search and look at Windows Event Log data.

[printul77700]

And what did you find out in that manual please ?
thanks

[kermitshort]

Thanks for your feedback, Masa. I haven't had time yet to evaluate the certificates. In the throes of troubleshooting, I did look at the supported protocols and versions for openssl on each system. I think I remember that they were close enough that it wouldn't cause any problems.

A question to the community at large, has anyone had any success using an Microsoft Active Directory integrated Certificate Authority to sign CSRs for use with forwarder-indexer communication encryption? Would you mind sharing the process you used to sign such certificates? My primary concern is that the certificates coming from my MS CA 1) don't include keys, and 2) must be converted from DER to PEM. While I did append the key files in regards to 1), I'm not sure that I've even made the key file correctly. I've also seen info in the documentation that indicates password protected keys are not supported, and that gives me pause. I'm hoping someone has already surmounted these challenges and questions. Any technical details or processes you can recommend would be appreciated. At this point I'm considering bypassing the MS CA and keeping everything in Linux, but that's not my preferred path.

[kermitshort]

OK so after doing all the certificates through the OpenSSL CA, I am seeing a lot less errors in log files, but I'm still not sure I'm getting data from my windows universal forwarder. In the web interface, when I look at the information under Settings > Data Inputs, all I see are local inputs on the indexer. I don't see any Forwarded inputs (All Forwarded Input Types are shown as 0, including the Windows Event Logs). I'm so new to Splunk, I'm not sure how to search for actual Windows Event Log data to see if I'm getting anything. What I see in the web interface makes me think that I'm still not connected, but I'm so green, I don't know if my assumption is accurate. Any advice?

[Masa · Accepted solution]

typo sslVersions = tls, =tls1.0 ?

unknown protocol error could be caused by the Splunk instance could not read or verify certificate or protocol from forwarder didn't match, or something else 🙂

1. Splunk instance could not read or verify certificate => in splunkd.log, warn or error message for reading or verifying SSL certificate or private key. This case, you need to re-visit configuration and certificate. I usually use openssl commands to verify all pem files and check if I can start a server using openssl s_server option.
2. forwarder's ssl version didn't match => I've seen this error when ssl client is trying to use sslv2 while ssl server is not accepting. So, ssl supported version compatibility could be cause of the issue.

If both of the above is okay, I would try the configuration without sslCommonNameToCheck, sslVerifyServerCert and sslVersions (as a common troubleshooting procedure, to make sure start with simplest settings works)

Other useful link to review SSL configuration is;
https://answers.splunk.com/answers/7164/how-do-i-set-up-ssl-forwarding-with-new-self-signed-certific...

[kermitshort]

I'm accepting these answers, as I am not getting any errors in the configuration file, but I'm still not sure I'm actually sucking in data from my Universal Forwarder.

[kermitshort]

OK I can see Windows Event Log data now. I had to read about the Splunk TA for Windows and how to use it. Thanks for all the help!

[Masa]

cool! Thank you for adding your troubleshooting steps 🙂

[Masa]

A question to the community at large,
has anyone had any success using an
Microsoft Active Directory integrated
Certificate Authority to sign CSRs for
use with forwarder-indexer
communication encryption?

Sorry but I do not have detail for procedure. But, I helped a user for this in the past through Support case. Two things are different topic, 1. How to create pem certificate based on ADCA. Whatever a CA server is, you must be able to collect certificate, CA cert, your own private key for pem format. This is the first step before Splunk configuration. Once first step is correctly done, second step is to them together to meet Splunk configuration. For Splunk configuration for SSL settings, follow the configurations step by step.

Again, it is a little bit difficult to troubleshoot for this question step-by-step in this Splunk Answer. But, there are good answers for many SSL setting problems in this community, in addition to Splunk doc. If none of them are helpful, I recommend to file a Support case.

[kermitshort]

Thanks, Masa. I believe I have done everything you wrote about. Maybe I can document it below as a starting point for others, as well for discussion:

1. Create a Private Server Key (using Openssl on Indexer Server) [Output in PEM format]
2. Create a Certificate Signing Request based on the Private Server Key [Output in PEM Format]
3. Using a web browser, submit the signing request to the Microsoft Certificate Authority
4. Once request is approved and certificate is signed by the CA, download the signed certificate [Output in DER Format]
5. Download the CA root certificate [Output in DER Format]
6. Using OpenSSL, convert the signed certificate from DER to PEM format [Output in PEM Format]
7. Using OpenSSL, conver the root CA certificate from DER to PEM format [Output in PEM Format]
8. Copy first the signed server cert to a new file, and append the private server key to the new file [Output in PEM Format]
9. Configure inputs.conf to point to the combined certificate as well as the root CA certificate (and specify any passwords to the private server key)

Perform steps 1-9 again to create a certificate that will be used on all forwarders.

Unfortunately, even after doing all of this, I still get the Protocol Unknown error.

My next attempt will be to create a local CA on the Indexer using OpenSSL instead of using the Microsoft CA. Any suggestions are welcome!