Deployment Architecture

Splunk Self-signed Certificate Error. Error setting up SSL for TCP data input from file=inputs.conf stanza="SSL"

azer271
Explorer

Hello. Im new at Splunk. Recently, I am trying to create and sign my own TLS certificates, following this official guide. https://docs.splunk.com/Documentation/Splunk/9.2.1/Security/Howtoself-signcertificates

However, splunkd.log keep on showing this error:

Error setting up SSL for TCP data input from file=inputs.conf stanza="SSL": Can't read key file /opt/splunk/etc/auth/mycerts/myServerCertificate.pem SSL error code=151441516 message="error:0906D06C:PEM routines:PEM_read_bio:no start line"

azer271_1-1715692419191.pngazer271_2-1715692434292.png

 

First, By following the guide, I created:

private key of root certificate authority certificate, which is myCertAuthPrivateKey.key

CSR for the certificate, which is myCertAuthCertificate.csr

root certificate authority certificate, which is myCertAuthCertificate.pem

Moreover, I created a server certificate and sign them with the root certificate authority certificate.

private key for the server certificate, which is myServerPrivateKey.key

CSR for the server certificate, which is myServerCertificate.csr

Server certificate, which is myServerCertificate.pem

 

Basically, following the guide, i have 6 files in mycerts folder, and one srl file. This Splunk Master is a master node connects to 3 indexers (clustering). I followed this guide to modify the configuration files, which is the inputs.conf and server.conf i believe.

Ref:

https://docs.splunk.com/Documentation/Splunk/9.2.1/Security/ConfigureSplunkforwardingtousesignedcert...

6+1 files for certificate.

azer271_0-1715691746708.png

/opt/splunk/etc/system/local/server.conf

[general]
...
[sslConfig]
sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCertAuthCertificate.pem
sslPassword = mypassword

...

/opt/splunk/etc/system/local/inputs.conf

[splunktcp-ssl:9997]
disabled=0

[SSL]
serverCert = /opt/splunk/etc/auth/mycerts/myServerCertificate.pem
sslPassword = mypassword
requireClientCert = true
sslVersions = *,-ssl2

Everytime i do service splunk restart, i still get the SSL error. Anyone know why and whats happening?? Same error is also happening in any other indexes. (same steps as i mentioned above)

0 Karma

deepakc
Builder

From the screenshot of files permissions  - the files look like they are for root, set those for the splunk user and try that  

azer271
Explorer

Hmm. I still get the same error. 😞

Certs permissions:

azer271_2-1715694526777.png

 

azer271_1-1715694513976.png

After restart,

Splunkd.log:

azer271_0-1715694381724.png

 

0 Karma

deepakc
Builder

Try a couple of things

Add the below to inputs.conf  - restart (YOUR CA CERT) under [SSL]

rootCA = /opt/splunk/etc/auth/mycerts/myCertAuthCertificate.pem


Run these to validate the certs - see if they read and show information

openssl rsa -in /opt/splunk/etc/auth/mycerts/myServerCertificate.pem -text

openssl x509 -in /opt/splunk/etc/auth/mycerts/myServerCertificate.pem -text -noout

 

azer271
Explorer

I fixed the error of "Can't read key file" by putting the contents of my server private key into the pem file.

Using this two commands can properly show information now:

openssl rsa -in /opt/splunk/etc/auth/mycerts/myServerCertificate.pem -text

openssl x509 -in /opt/splunk/etc/auth/mycerts/myServerCertificate.pem -text -noout

openssl rsa is properly showing the rsa private key (modulus, prime etcetc) now. openssl x509 works fine as i mentioned before.

However, splunkd.log still shows sslv3 alert certificate unknown.

azer271_0-1716128688633.png

Thanks.

 

0 Karma

deepakc
Builder

So this initially looks like the sender does not have certs, what is 192.168.100.1? (The client sending should now have the TLS certs - what does the outputs from client (UF ) look like?


Test from the client
openssl s_client -connect <hostname>:9997

Or 

/opt/splunkforwarder/bin/splunk cmd openssl s_client -connect <hostname>:9997

0 Karma

azer271
Explorer

Hello. Sorry about the late reply. After adding the rootCA setting, it still does not work. However, openssl shows "unable to load private key" ,which I believe this may be the issue. Regenerating the certs/keys also have the same issue.🤔

Here is the output of openssl:

The private key is unable to load.

azer271_1-1715853086798.png

The cert is showed properly.

azer271_0-1715853053511.png

Morever, the search result shows that the ssl is still false. (I set up certs in hf and forwarder for testing)

Troubleshoot output:

azer271_3-1715853521831.png

Thank you for your help btw. 

0 Karma

deepakc
Builder

Hi @azer271 

Have a look at this Splunk TLS config page. It sounds like there's a step / config missing, work through this and your steps.   That error could be  incorrect PEM format  or  some config settings  

https://lantern.splunk.com/Splunk_Platform/Product_Tips/Administration/Securing_the_Splunk_platform_... 

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

The "no_start_line" error suggests format mismatch.

Proper PEM-formatted cert or key file should begin with a header.

See https://en.wikipedia.org/wiki/Privacy-Enhanced_Mail

 

0 Karma
Get Updates on the Splunk Community!

App Building 101 - Build Your First App!

WATCH RECORDING NOW   Tech Talk: App Dev Edition Splunk has tons of out-of-the-box functionality, and you’ve ...

Introducing support for Amazon Data Firehose in Splunk Edge Processor

We’re excited to announce a powerful update to Splunk Data Management with added support for Amazon Data ...

The Observability Round-Up: September 2024

What’s up Splunk Community! Welcome to the latest edition of the Observability Round-Up, a monthly series in ...