topic Re: Splunk Self-signed Certificate Error. Error setting up SSL for TCP data input from file=inputs.conf stanza="SSL in Deployment Architecture

Splunk Self-signed Certificate Error. Error setting up SSL for TCP data input from file=inputs.conf stanza="SSL"

azer271 — Tue, 14 May 2024 13:36:24 GMT

Hello. Im new at Splunk. Recently, I am trying to create and sign my own TLS certificates, following this official guide. https://docs.splunk.com/Documentation/Splunk/9.2.1/Security/Howtoself-signcertificates

However, splunkd.log keep on showing this error:

Error setting up SSL for TCP data input from file=inputs.conf stanza="SSL": Can't read key file /opt/splunk/etc/auth/mycerts/myServerCertificate.pem SSL error code=151441516 message="error:0906D06C:PEM routines:PEM_read_bio:no start line"

First, By following the guide, I created:

private key of root certificate authority certificate, which is myCertAuthPrivateKey.key

CSR for the certificate, which is myCertAuthCertificate.csr

root certificate authority certificate, which is myCertAuthCertificate.pem

Moreover, I created a server certificate and sign them with the root certificate authority certificate.

private key for the server certificate, which is myServerPrivateKey.key

CSR for the server certificate, which is myServerCertificate.csr

Server certificate, which is myServerCertificate.pem

Basically, following the guide, i have 6 files in mycerts folder, and one srl file. This Splunk Master is a master node connects to 3 indexers (clustering). I followed this guide to modify the configuration files, which is the inputs.conf and server.conf i believe.

Ref:

https://docs.splunk.com/Documentation/Splunk/9.2.1/Security/ConfigureSplunkforwardingtousesignedcertificates

6+1 files for certificate.

/opt/splunk/etc/system/local/server.conf

[general]
...

[sslConfig]
sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCertAuthCertificate.pem
sslPassword = mypassword

...

/opt/splunk/etc/system/local/inputs.conf

[splunktcp-ssl:9997]
disabled=0

[SSL]
serverCert = /opt/splunk/etc/auth/mycerts/myServerCertificate.pem
sslPassword = mypassword
requireClientCert = true
sslVersions = *,-ssl2

Everytime i do service splunk restart, i still get the SSL error. Anyone know why and whats happening?? Same error is also happening in any other indexes. (same steps as i mentioned above)

Re: Splunk Self-signed Certificate Error. Error setting up SSL for TCP data input from file=inputs.conf stanza="SSL

deepakc — Tue, 14 May 2024 13:38:32 GMT

From the screenshot of files permissions - the files look like they are for root, set those for the splunk user and try that

Re: Splunk Self-signed Certificate Error. Error setting up SSL for TCP data input from file=inputs.conf stanza="SSL

azer271 — Tue, 14 May 2024 13:49:20 GMT

Hmm. I still get the same error. 😞

Certs permissions:

After restart,

Splunkd.log:

Re: Splunk Self-signed Certificate Error. Error setting up SSL for TCP data input from file=inputs.conf stanza="SSL

deepakc — Tue, 14 May 2024 14:20:22 GMT

Try a couple of things

Add the below to inputs.conf - restart (YOUR CA CERT) under [SSL]

rootCA = /opt/splunk/etc/auth/mycerts/myCertAuthCertificate.pem

Run these to validate the certs - see if they read and show information

openssl rsa -in /opt/splunk/etc/auth/mycerts/myServerCertificate.pem -text openssl x509 -in /opt/splunk/etc/auth/mycerts/myServerCertificate.pem -text -noout

Re: Splunk Self-signed Certificate Error. Error setting up SSL for TCP data input from file=inputs.conf stanza="SSL

azer271 — Thu, 16 May 2024 09:59:01 GMT

Hello. Sorry about the late reply. After adding the rootCA setting, it still does not work. However, openssl shows "unable to load private key" ,which I believe this may be the issue. Regenerating the certs/keys also have the same issue.🤔

Here is the output of openssl:

The private key is unable to load.

The cert is showed properly.

Morever, the search result shows that the ssl is still false. (I set up certs in hf and forwarder for testing)

Troubleshoot output:

Thank you for your help btw.

Re: Splunk Self-signed Certificate Error. Error setting up SSL for TCP data input from file=inputs.conf stanza="SSL

deepakc — Thu, 16 May 2024 10:13:40 GMT

Hi @azer271

Have a look at this Splunk TLS config page. It sounds like there's a step / config missing, work through this and your steps. That error could be incorrect PEM format or some config settings

https://lantern.splunk.com/Splunk_Platform/Product_Tips/Administration/Securing_the_Splunk_platform_with_TLS

Re: Splunk Self-signed Certificate Error. Error setting up SSL for TCP data input from file=inputs.conf stanza="SSL

PickleRick — Thu, 16 May 2024 10:31:55 GMT

The "no_start_line" error suggests format mismatch.

Proper PEM-formatted cert or key file should begin with a header.

See https://en.wikipedia.org/wiki/Privacy-Enhanced_Mail

Re: Splunk Self-signed Certificate Error. Error setting up SSL for TCP data input from file=inputs.conf stanza="SSL

azer271 — Sun, 19 May 2024 14:25:42 GMT

I fixed the error of "Can't read key file" by putting the contents of my server private key into the pem file.

Using this two commands can properly show information now:

openssl rsa -in /opt/splunk/etc/auth/mycerts/myServerCertificate.pem -text

openssl x509 -in /opt/splunk/etc/auth/mycerts/myServerCertificate.pem -text -noout

openssl rsa is properly showing the rsa private key (modulus, prime etcetc) now. openssl x509 works fine as i mentioned before.

However, splunkd.log still shows sslv3 alert certificate unknown.

Thanks.

Re: Splunk Self-signed Certificate Error. Error setting up SSL for TCP data input from file=inputs.conf stanza="SSL

deepakc — Mon, 20 May 2024 17:12:58 GMT

So this initially looks like the sender does not have certs, what is 192.168.100.1? (The client sending should now have the TLS certs - what does the outputs from client (UF ) look like?

Test from the client
openssl s_client -connect <hostname>:9997

/opt/splunkforwarder/bin/splunk cmd openssl s_client -connect <hostname>:9997