Why is SSL not working on our Splunk 6.3.0 Windows universal forwarder with error "SSL clause not found or servercert not provided"?

[joshuabiggley]

We've been trying to get the Splunk Universal Forwarder for Windows (v6.3.0) to work on a Windows 2008 R2 server and we consistently get the following error.

TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available

We turned on debug logs and saw a little more detail but we're still having issues.

02-15-2016 12:42:55.522 -0600 DEBUG TcpOutputProc - Found group : splunkssl
02-15-2016 12:42:55.522 -0600 DEBUG TcpOutputProc - confifuring ssl for cert path :D:/Program Files/SplunkUniversalForwarder/etc/auth/server.pem
02-15-2016 12:42:55.522 -0600 INFO  TcpOutputProc - tcpout group splunkssl using Auto load balanced forwarding
02-15-2016 12:42:55.522 -0600 INFO  TcpOutputProc - Group splunkssl initialized with maxQueueSize=512000 in bytes.

First, we've tried all sorts of iterations for the .pem file paths in the outputs.conf file. (We are using the D:\Program Files\SplunkUniversalForwarder\etc\system\local\outputs.conf file). This is what the current version looks like, but we've tried lots of different iterations. (quoted, unquoted, double backslash //)

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
server = X.X.X.X:9997
sslRootCAPath = D:/Program Files/SplunkUniversalForwarder/etc/auth/cacert.pem
sslCertPath = D:/Program Files/SplunkUniversalForwarder/etc/auth/server.pem
sslPassword = {encrypted text removed}
sslVerifyServerCert = true

We are using self-signed certificates, but we found that we had to rename them to cacert.pem and server.pem or else we generated a completely different error.

02-15-2016 10:01:50.425 -0600 ERROR SSLCommon - Can't read key file D:\Program Files\SplunkUniversalForwarder\etc\auth\server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.

I expect that someone has this working. Any Windows-specific recommendations?

[joshuabiggley]

The output from our splunkd.log file for the SSL transaction. Here's what we've got:

02-16-2016 10:23:26.556 -0600 INFO  TcpOutputProc - Initializing with fwdtype=lwf
02-16-2016 10:23:26.571 -0600 DEBUG TcpOutputProc - Setting : negotiateNewProtocol=true
02-16-2016 10:23:26.571 -0600 DEBUG TcpOutputProc - Setting : channelReapInterval=60000
02-16-2016 10:23:26.571 -0600 DEBUG TcpOutputProc - Setting : channelTTL=300000
02-16-2016 10:23:26.571 -0600 DEBUG TcpOutputProc - Setting : channelReapLowater=10
02-16-2016 10:23:26.571 -0600 DEBUG TcpOutputProc - Setting : dnsResolutionInterval=300
02-16-2016 10:23:26.571 -0600 DEBUG TcpOutputProc - Setting : useClientSSLCompression=true
02-16-2016 10:23:26.571 -0600 DEBUG TcpOutputProc - Mymap is: PropertiesMap: {ackTimeoutOnShutdown -> '30' autoLBFrequency -> '30' blockOnCloning -> 'true' blockWarnThreshold -> '100' channelReapInterval -> '60000' channelReapLowater -> '10' channelTTL -> '300000' compressed -> 'false' connectionTimeout -> '20' defaultGroup -> 'splunkssl' disabled -> 'false' dnsResolutionInterval -> '300' dropClonedEventsOnQueueFull -> '5' dropEventsOnQueueFull -> '-1' forceTimebasedAutoLB -> 'false' forwardedindex.0.whitelist -> '.*' forwardedindex.1.blacklist -> '_.*' forwardedindex.2.whitelist -> '(_audit|_introspection)' forwardedindex.filter.disable -> 'false' heartbeatFrequency -> '30' indexAndForward -> 'false' maxConnectionsPerIndexer -> '2' maxFailuresPerInterval -> '2' maxQueueSize -> 'auto' negotiateNewProtocol -> 'true' readTimeout -> '300' secsInFailureInterval -> '1' sendCookedData -> 'true' sslQuietShutdown -> 'false' tcpSendBufSz -> '0' useACK -> 'false' useClientSSLCompression -> 'true' writeTimeout -> '300'}
02-16-2016 10:23:26.571 -0600 INFO  TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
02-16-2016 10:23:26.571 -0600 INFO  TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
02-16-2016 10:23:26.571 -0600 INFO  TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
02-16-2016 10:23:26.571 -0600 DEBUG TcpOutputProc - None of whitelist/blacklist for : 3 not found
02-16-2016 10:23:26.571 -0600 DEBUG TcpOutputProc - Found group : splunkssl
02-16-2016 10:23:26.571 -0600 DEBUG TcpOutputProc - confifuring ssl for cert path :D:\Program Files\SplunkUniversalForwarder\etc\auth\server.pem
02-16-2016 10:23:26.571 -0600 DEBUG TcpOutputProc - Key file password requires decrypting
02-16-2016 10:23:26.571 -0600 INFO  TcpOutputProc - tcpout group splunkssl using Auto load balanced forwarding
02-16-2016 10:23:26.571 -0600 INFO  TcpOutputProc - Group splunkssl initialized with maxQueueSize=512000 in bytes.
02-16-2016 10:23:26.571 -0600 DEBUG TcpOutputProc - Will resolve indexer names at 300 second interval.
02-16-2016 10:23:26.571 -0600 DEBUG TcpOutputProc - IndexAndForward explicitly set to : false
02-16-2016 10:23:26.571 -0600 DEBUG TcpOutputProc - ACK timeout during shutdown = 30
02-16-2016 10:23:26.571 -0600 INFO  PipelineComponent - Pipeline merging disabled in default-mode.conf file
02-16-2016 10:23:26.571 -0600 INFO  PipelineComponent - Pipeline typing disabled in default-mode.conf file
02-16-2016 10:23:26.571 -0600 INFO  PipelineComponent - Pipeline vix disabled in default-mode.conf file
02-16-2016 10:23:26.634 -0600 INFO  TcpInputProc - Registering metrics callback for: tcpin_connections
02-16-2016 10:23:26.634 -0600 INFO  TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available

This is with the following entries in our outputs.conf file:

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
server = X.X.X.X:9997
requireClientCert = false
sslRootCAPath = D:\Program Files\SplunkUniversalForwarder\etc\auth\cacert.pem
sslCertPath = D:\Program Files\SplunkUniversalForwarder\etc\auth\server.pem
sslPassword = {encrypted text here}

[michael_schmidt · Accepted solution]

First off, call support. I beat my head against this wall for about a week, then they fixed me up in about 2 hours. Save your head the beating.

Couple things for you to try if you're determined to beat your head against this wall. Open up your Certs (plug for Notepad ++ here) and take out any of the filler stuff i.e.:

Bag Attributes
localKeyID: 01 00 00 00
friendlyName: le-4bd84fed-9564-4df3-beed-1f4aac732c07
Microsoft CSP Name: Microsoft RSA SChannel Cryptographic Provider
Key Attributes
X509v3 Key Usage: 10

Your Certs should look like this when you're done:

-----BEGIN PRIVATE KEY-----
EncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbage
EncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbage
EncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbage
EncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbage
EncryptionGarbageEncryptionGarbage
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
EncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbage
EncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbage
EncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbage
EncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbageEncryptionGarbage
EncryptionGarbageEncryptionGarbageEncryptionGarbage
-----END CERTIFICATE-----

And nothing Else. You'll need to put together the Client cert and the Key together in the in one pem and that's the file you point to with sslCertPath = And One File that Has both the CertAuth PEM and the Client Cert together in one PEM is the sslRootCAPath = File.

i.e.
sslCertPath = D:/Program Files/SplunkUniversalForwarder/ClientCertAndClientKeyTogetherInOneFile.PEM

sslRootCAPath = D:/Program Files/SplunkUniversalForwarder/etc/auth/ClientCertAndCACertInOneFile.PEM

You'll need a similar setup on the Indexer/Forwarder that you're sending to. In your $SPLUNK_HOME/etc/system/local/Inputs.conf

[splunktcp-ssl:9998]
compressed = true

[SSL]
password = PASSWORD-GOES-HERE
requireClientCert = false <<----- Trust me on this one, even support couldn't make this work with TRUE in their own environment.
rootCA = $SPLUNK_HOME\etc\auth\IndexerServerCertandRootCACertINOneFile.pem
serverCert = $SPLUNK_HOME\etc\auth\IndexerServerKeyAndServerCertINOneFile.pem

After that It should all work.

[SandzVG]

Hi Michael,

Thanks for your post, could you please share a bit more on how the certs are generated (Own) and the forwarder configuration.

[joshuabiggley]

Note the the actual fix was to rebuild and redistribute the certificates as per the link below, but this answer set us on the right path to come to that conclusion so is the accepted answer. Thanks!

https://answers.splunk.com/answers/7164/how-do-i-set-up-ssl-forwarding-with-new-self-signed-certific...

[michael_schmidt]

Glad you found the answer!

[s2_splunk]

See if this helps.

[joshuabiggley]

We've reviewed that document already. It is Linux-centric and we're looking for some Windows specific guidance. In reviewing it just now I did note this line in step #5

"The common name on the indexer's server certificate ("serverCert" in inputs.conf, as defined in step 4) matches one of the "sslCommonNameToCheck" settings."

According to the outputs.conflink text this isn't a mandatory field but it is the only item I could see that we didn't have in our outputs, though we did specify sslVerifyServerCert = true. I did remove that entry and tested, but it did not resolve our issue. We are still getting the 'SSL clause not found' entries in the splunkd.log file.

[joshuabiggley]

I accepted the answer from @michael_schmidt because it was more detailed and ultimately led me down the right path, but your link was what we followed. We ended up ditching our existing certs based on a suspicion that they were not built correctly. Even though Linux UFs passed data without an issue, the Windows boxes wouldn't play nice.

When we followed the steps again we were able to get all of our Windows UFs in our sandbox up and running.

Thanks!

[s2_splunk]

I don't think that there are any differences in the way this needs to be setup between Windows and Linux. Your error message in the original post said "Bad decrypt", which indicates that the hashed password is not valid.
What is that other error messages about "SSL clause not found" exactly saying, are you able to post the consecutive log entries from start of SSL initialization to error message?