Splunk v6.3 running on Windows Server 2008 R2
Server is a SearchHead
I am getting the socket errors pretty much non stop after SSL/TLS'ing everything.
https://localhost:8000 comes up fine and I can log in.
https://localhost:8089 comes up fine
Here are my configs:
ssl area from SERVER.CONF:
[sslConfig] caCertFile = chain.pem caPath = $SPLUNK_HOME/etc/auth/DOD cipherSuite = HIGH sslKeysfile = private_password.pem sslKeysfilePassword = **HASHEDPASSWORD** sslVersions = tls1.1, tls1.2
[tcpout] defaultGroup = myindexers [tcpout:myindexers] compressed = true server = server1:9997,server2:9997 sslCertPath = $SPLUNK_HOME/etc/auth/DOD/private_password.pem sslPassword = **HASHEDPASSWORD** sslRootCAPath = $SPLUNK_HOME/etc/auth/DOD/chain.pem sslVerifyServerCert = false useACK = true
[settings] enableSplunkWebSSL = true privKeyPath = etc/auth/splunkweb/web_private.pem caCertPath = etc/auth/splunkweb/web_chain.pem sslVersions = tls1.1, tls1.2
EXTRACTED FROM SPLUNKD.LOG
10-26-2015 11:56:49.705 -0400 INFO TcpOutputProc - Connected to idx=server2:9997 using ACK. 10-26-2015 12:03:09.391 -0400 WARN HttpListener - Socket error from 127.0.0.1 while idling: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol 10-26-2015 12:04:08.737 -0400 WARN HttpListener - Socket error from 127.0.0.1 while idling: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol 10-26-2015 12:04:18.947 -0400 INFO TcpOutputProc - Connected to idx=server1:9997 using ACK. 10-26-2015 12:04:20.213 -0400 WARN HttpListener - Socket error from 127.0.0.1 while idling: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol 10-26-2015 12:04:32.094 -0400 WARN HttpListener - Socket error from 127.0.0.1 while idling: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol 10-26-2015 12:04:44.570 -0400 WARN HttpListener - Socket error from 127.0.0.1 while idling: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
when I double click on my public certificate, I see:
issued to: myservers FQDN
issued by: subordinate
expires in about two years
My private_password.pem file is my private key exported in openssl format with a password, begins with:
-----BEGIN RSA PRIVATE KEY-----
My chain.pem is built as
My web_private.pem is my private key exported in openssl format without a password
I was receiving the same issue on a new Splunk instance with almost identical conf files as you. I also received the same socket error when my forwarders specified the cipherSuite server.conf. This went away when I commented out the cipherSuite line in server.conf on the forwarder side. Commenting out the cipherSuite line on the server side also resolves the socket errors in splunkd.log that you are seeing.
I see "dod" in the path to your certs, so I'm assuming that you have done FIPS_MODE=1 prior to starting Splunk. I guess this is a bug and you cannot use "cipherSuite" to force a cipher while also being FIPS compliant. FIPS will remove instances of SSL2/SSL3 regardless of what the CONFs say, so it's my assumption that the cipherSuite key is redundant and is causing issues.
SPL-92435 - Forcing TLS1.2 or TLS1.1 in server.conf with SPLUNK_FIPS does not work.
I checked the $SPLUNK_HOME/etc/splunk-launch.conf and there is no entry for FIPS_MODE=1.
I changed my server.conf file to:
sslKeysfilePassword = HASHEDPASSWORD
errors went away and now server is using the default splunk cert, vice my DOD cert.
Might you have any experience/guidance on using the DOD keys for this stanza:
sslKeysfile = private_password.pem - - is this just openssl formatted PEM key with password?
I cannot get splunk to read my private key. I keep getting errors: pem_read_bio:no start line
The sslKeysFile is your certificate chain formatted in: server.pem > serverPrivateKey.key > yourCAPublicKey.pem where "server.pem" is your certificate signed by your CA.
Make sure you create these using a command prompt or terminal (removing the possibility of any Unicode characters in the file).
type server.pem serverPrivateKey.key yourCAPublicKey.pem > myCertificateChain.pem
cat server.pem serverPrivateKey.key yourCAPublicKey.pem > myCertificateChain.pem
I specified my certificate chain in server.conf and left the sslKeysFilePassword setting alone. I did find I had issues with the certificate chain being in a folder deeper than /etc/auth (slow web server startup and sometimes it would just stop), so I left it in that folder.
[sslConfig] allowSslCompression = false caCertFile = myCertificateChain.pem caPath = $SPLUNK_HOME/etc/auth sslKeysfilePassword = HASHEDPASSWORD sslVersions = *,-ssl2,-ssl3
Your /etc/system/local/web.conf should be:
[settings] enableSplunkWebSSL = 1 sslVersions = *,-ssl2,-ssl3 httpport = 443 privKeyPath = etc/auth/mycerts/serverPrivateKey.key caCertPath = etc/auth/mycerts/server.pem
I followed these directions for mine: