Failed to contact license manager: reason='Unable to connect to license manager=https:// Error connecting: error:1409008

[Gregski11]

Failed to contact license manager: reason='Unable to connect to license manager=https://SplunkInstance01.MyDomain.net:8089

Error connecting:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed -

please check the output of the `openssl verify` command for the certificates involved;

note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name.', first failure time= [...]

first mea culpa for asking this as I am sure it has been asked before but I couldn't quite understand how to use the openssl verify command, when I try to run it I get this error:

C:\Program Files\Splunk\etc\auth\distServerKeys>openssl verify -CAfile private.pem trusted.pem

WARNING: can't open config file: C:\\jnkns\\workspace\\build-home/ssl/openssl.cnf
Error loading file private.pem

I also tried to run it from the bin subdirectory, home of the openssl utility 

C:\Program Files\Splunk\bin>openssl verify -CAfile "C:\Program Files\Splunk\etc\auth\distServerKeys\private.pem" "C:\Program Files\Splunk\etc\auth\distServerKeys\trusted.pem"

WARNING: can't open config file: C:\\jnkns\\workspace\\build-home/ssl/openssl.cnf
Error loading file C:\Program Files\Splunk\etc\auth\distServerKeys\private.pem

I suspect this private \ public key pair combination may still be the stale default self signed combination cause Splunk to frown upon it, however what is perplexing to me is that it works on the other 20 plus servers, so I throw yourself upon your mercy for help

please note we have over 2 dozen Splunk servers running version 9.0.0 all on Windows platforms and this is the only server getting this error, all servers use our own Microsoft CA internal Enterprise certificates based on a two tier (RootCA \ IntermediateCA architecture) so I think I know what I am doing ha ha at least in terms of certing  

[Gregski11 · Accepted solution]

thank you splunkd.log 

01-05-2023 23:03:47.881 -0800 WARN SSLOptions [0 MainThread] - server.conf/[sslConfig]/sslRootCAPath (C:\Program Files\Splunk\etc\auth\myCertificateChain.pem): does not exist; may not be able to use SSL

the actual path is:

C:\Program Files\Splunk\etc\auth\mycerts\myCertificateChain.pem

[SanjayReddy]

Hi @Gregski11 

Just a try

can you try below 
comment sslPassword in etc-->system-->local-->server.conf  use # before sslPassword

[sslConfig]
#sslPassword = $7$oLfcw7xnJJyWM8y99QD1E0oXOpUvIdakzT8BHm2cpWaJrJB5U6yKZQ==

and restart the splunk ?.

[Gregski11]

thank you Sanjay for offering to help, I tried what you just said and the Splunk Restart just sits at the following command

Waiting for web server at https://127.0.0.1:8000 to be available....

What I don't understand is how or why the "Server" certificate impacts Splunk Web from starting when I am using a second and completely separate SSL web certificate for Splunk Web, that doesn't make any sense to me

also as mentioned before I have 21 other Splunk servers where I did the exact precise same setup/configuration and those work fine, here's what lurks inside all of our server.conf files, they are all the same on purpose to make our configuration and administration consistent and uniform (but yes the passwords use each servers unique salt etc to hash the passwords)

[sslConfig]
serverCert = C:\Program Files\Splunk\etc\auth\myServerCertificate.pem
sslRootCAPath = C:\Program Files\Splunk\etc\auth\myCertificateChain.pem
sslPassword = $7$AfXR3titQZ6t0CxjunksAndStuffZN8dJunkbHStuffvyU7lX4=
cliVerifyServerName = true