Splunk Self-signed Certificate Error. Error setting up SSL for TCP data input from file=inputs.conf stanza="SSL"

[azer271]

Hello. Im new at Splunk. Recently, I am trying to create and sign my own TLS certificates, following this official guide. https://docs.splunk.com/Documentation/Splunk/9.2.1/Security/Howtoself-signcertificates

However, splunkd.log keep on showing this error:

Error setting up SSL for TCP data input from file=inputs.conf stanza="SSL": Can't read key file /opt/splunk/etc/auth/mycerts/myServerCertificate.pem SSL error code=151441516 message="error:0906D06C:PEM routines:PEM_read_bio:no start line"

First, By following the guide, I created:

private key of root certificate authority certificate, which is myCertAuthPrivateKey.key

CSR for the certificate, which is myCertAuthCertificate.csr

root certificate authority certificate, which is myCertAuthCertificate.pem

Moreover, I created a server certificate and sign them with the root certificate authority certificate.

private key for the server certificate, which is myServerPrivateKey.key

CSR for the server certificate, which is myServerCertificate.csr

Server certificate, which is myServerCertificate.pem

Basically, following the guide, i have 6 files in mycerts folder, and one srl file. This Splunk Master is a master node connects to 3 indexers (clustering). I followed this guide to modify the configuration files, which is the inputs.conf and server.conf i believe.

Ref:

https://docs.splunk.com/Documentation/Splunk/9.2.1/Security/ConfigureSplunkforwardingtousesignedcert...

6+1 files for certificate.

/opt/splunk/etc/system/local/server.conf

[general]
...

[sslConfig]
sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCertAuthCertificate.pem
sslPassword = mypassword

...

/opt/splunk/etc/system/local/inputs.conf

[splunktcp-ssl:9997]
disabled=0

[SSL]
serverCert = /opt/splunk/etc/auth/mycerts/myServerCertificate.pem
sslPassword = mypassword
requireClientCert = true
sslVersions = *,-ssl2

Everytime i do service splunk restart, i still get the SSL error. Anyone know why and whats happening?? Same error is also happening in any other indexes. (same steps as i mentioned above)

[deepakc]

From the screenshot of files permissions  - the files look like they are for root, set those for the splunk user and try that  

[azer271]

Hmm. I still get the same error. 😞

Certs permissions:

After restart,

Splunkd.log:

[deepakc]

Try a couple of things

Add the below to inputs.conf  - restart (YOUR CA CERT) under [SSL]

rootCA = /opt/splunk/etc/auth/mycerts/myCertAuthCertificate.pem

Run these to validate the certs - see if they read and show information

openssl rsa -in /opt/splunk/etc/auth/mycerts/myServerCertificate.pem -text

openssl x509 -in /opt/splunk/etc/auth/mycerts/myServerCertificate.pem -text -noout

[azer271]

I fixed the error of "Can't read key file" by putting the contents of my server private key into the pem file.

Using this two commands can properly show information now:

openssl rsa -in /opt/splunk/etc/auth/mycerts/myServerCertificate.pem -text

openssl x509 -in /opt/splunk/etc/auth/mycerts/myServerCertificate.pem -text -noout

openssl rsa is properly showing the rsa private key (modulus, prime etcetc) now. openssl x509 works fine as i mentioned before.

However, splunkd.log still shows sslv3 alert certificate unknown.

Thanks.

[deepakc]

So this initially looks like the sender does not have certs, what is 192.168.100.1? (The client sending should now have the TLS certs - what does the outputs from client (UF ) look like?

Test from the client
openssl s_client -connect <hostname>:9997

Or 

/opt/splunkforwarder/bin/splunk cmd openssl s_client -connect <hostname>:9997

[azer271]

Hello. Sorry about the late reply. After adding the rootCA setting, it still does not work. However, openssl shows "unable to load private key" ,which I believe this may be the issue. Regenerating the certs/keys also have the same issue.🤔

Here is the output of openssl:

The private key is unable to load.

The cert is showed properly.

Morever, the search result shows that the ssl is still false. (I set up certs in hf and forwarder for testing)

Troubleshoot output:

Thank you for your help btw. 

[deepakc]

Hi @azer271 

Have a look at this Splunk TLS config page. It sounds like there's a step / config missing, work through this and your steps.   That error could be  incorrect PEM format  or  some config settings  

https://lantern.splunk.com/Splunk_Platform/Product_Tips/Administration/Securing_the_Splunk_platform_... 

[PickleRick]

The "no_start_line" error suggests format mismatch.

Proper PEM-formatted cert or key file should begin with a header.

See https://en.wikipedia.org/wiki/Privacy-Enhanced_Mail