Deployment Architecture
Highlighted

Why am I getting WARN httplistener socket errors in my splunkd.log on my Search Head server?

Path Finder

Splunk v6.3 running on Windows Server 2008 R2
Server is a SearchHead
I am getting the socket errors pretty much non stop after SSL/TLS'ing everything.

https://localhost:8000 comes up fine and I can log in.
https://localhost:8089 comes up fine

Here are my configs:

============================================================================
ssl area from SERVER.CONF:

[sslConfig]
caCertFile = chain.pem
caPath = $SPLUNK_HOME/etc/auth/DOD
cipherSuite = HIGH
sslKeysfile = private_password.pem
sslKeysfilePassword = **HASHEDPASSWORD**
sslVersions = tls1.1, tls1.2

============================================================================
OUTPUTS.CONF:

[tcpout]
defaultGroup = myindexers

[tcpout:myindexers]
compressed = true
server = server1:9997,server2:9997
sslCertPath = $SPLUNK_HOME/etc/auth/DOD/private_password.pem
sslPassword = **HASHEDPASSWORD**
sslRootCAPath = $SPLUNK_HOME/etc/auth/DOD/chain.pem
sslVerifyServerCert = false
useACK = true

============================================================================
WEB.CONF:

[settings]
enableSplunkWebSSL = true
privKeyPath = etc/auth/splunkweb/web_private.pem
caCertPath = etc/auth/splunkweb/web_chain.pem
sslVersions = tls1.1, tls1.2

============================================================================
EXTRACTED FROM SPLUNKD.LOG

10-26-2015 11:56:49.705 -0400 INFO  TcpOutputProc - Connected to idx=server2:9997 using ACK.
10-26-2015 12:03:09.391 -0400 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
10-26-2015 12:04:08.737 -0400 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
10-26-2015 12:04:18.947 -0400 INFO  TcpOutputProc - Connected to idx=server1:9997 using ACK.
10-26-2015 12:04:20.213 -0400 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
10-26-2015 12:04:32.094 -0400 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
10-26-2015 12:04:44.570 -0400 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

============================================================================
when I double click on my public certificate, I see:
issued to: myservers FQDN
issued by: subordinate
expires in about two years

Certificate path:

root -
intermediate -
subordinate -

myserver

My private_password.pem file is my private key exported in openssl format with a password, begins with:
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED

DEK-Info: DES-CBC.............

My chain.pem is built as
myserver-
subordinate-
intermediate-

root

My web_private.pem is my private key exported in openssl format without a password

My web_chain.pem is my public certificate

Highlighted

Re: Why am I getting WARN httplistener socket errors in my splunkd.log on my Search Head server?

Contributor

Hey rgsurfs!

I was receiving the same issue on a new Splunk instance with almost identical conf files as you. I also received the same socket error when my forwarders specified the cipherSuite server.conf. This went away when I commented out the cipherSuite line in server.conf on the forwarder side. Commenting out the cipherSuite line on the server side also resolves the socket errors in splunkd.log that you are seeing.

I see "dod" in the path to your certs, so I'm assuming that you have done FIPS_MODE=1 prior to starting Splunk. I guess this is a bug and you cannot use "cipherSuite" to force a cipher while also being FIPS compliant. FIPS will remove instances of SSL2/SSL3 regardless of what the CONFs say, so it's my assumption that the cipherSuite key is redundant and is causing issues.

Check out this post:
https://answers.splunk.com/answers/214531/trying-to-secure-splunk-browser-is-throwing-sslhan.html

SPL-92435 - Forcing TLS1.2 or TLS1.1 in server.conf with SPLUNK_FIPS does not work.

0 Karma
Highlighted

Re: Why am I getting WARN httplistener socket errors in my splunkd.log on my Search Head server?

Path Finder

I checked the $SPLUNKHOME/etc/splunk-launch.conf and there is no entry for FIPSMODE=1.

I changed my server.conf file to:

[sslConfig]
sslKeysfilePassword = HASHEDPASSWORD

errors went away and now server is using the default splunk cert, vice my DOD cert.

Might you have any experience/guidance on using the DOD keys for this stanza:
sslKeysfile = privatepassword.pem - - is this just openssl formatted PEM key with password?
I cannot get splunk to read my private key. I keep getting errors: pem
read_bio:no start line

0 Karma
Highlighted

Re: Why am I getting WARN httplistener socket errors in my splunkd.log on my Search Head server?

Contributor

The sslKeysFile is your certificate chain formatted in: server.pem > serverPrivateKey.key > yourCAPublicKey.pem where "server.pem" is your certificate signed by your CA.

Make sure you create these using a command prompt or terminal (removing the possibility of any Unicode characters in the file).

For Windows:
type server.pem serverPrivateKey.key yourCAPublicKey.pem > myCertificateChain.pem

For Linux:
cat server.pem serverPrivateKey.key yourCAPublicKey.pem > myCertificateChain.pem

I specified my certificate chain in server.conf and left the sslKeysFilePassword setting alone. I did find I had issues with the certificate chain being in a folder deeper than /etc/auth (slow web server startup and sometimes it would just stop), so I left it in that folder.

[sslConfig]
allowSslCompression = false
caCertFile = myCertificateChain.pem
caPath = $SPLUNK_HOME/etc/auth
sslKeysfilePassword = HASHEDPASSWORD
sslVersions = *,-ssl2,-ssl3

Your /etc/system/local/web.conf should be:

[settings]
enableSplunkWebSSL = 1
sslVersions = *,-ssl2,-ssl3
httpport = 443
privKeyPath = etc/auth/mycerts/serverPrivateKey.key
caCertPath = etc/auth/mycerts/server.pem

I followed these directions for mine:
http://docs.splunk.com/Documentation/Splunk/6.2.0/Security/HowtoprepareyoursignedcertificatesforSplu...

0 Karma
Highlighted

Re: Why am I getting WARN httplistener socket errors in my splunkd.log on my Search Head server?

Path Finder

So, my cacertfile and sslkeysfile are the same file ? format?

0 Karma