Need help with reconfiguring LM and DMC on rebuilt linux server

[sdkp03]

We did rebuild existing server that hosted LM and DMC. I did install latest splunk on the rebuilt server. Copied configuration settings from old build. Now when I try to turn  on splunk service, the service does start up fine but when I try accessing it from UI, I see below mentioned errors logged in splunkd.log:

Error in Indexer Discovery communication. Verify
that the pass4SymmKey set under [indexer_discovery:dr_indexer_cluster_group] in 'outputs.conf' matches the same setting
under [indexer_discovery] in 'server.conf' on the Cluster Master. [uri=https://splunk-idx-cm.trgtm.ostravam.corp.tels
tra.com:8089/services/indexer_discovery http_code=502 http_response="Error connecting: SSL not configured on client"]

I did decrypt and verified that the pass4symmkey is same on CM and the new server. 

> SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 err
or:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt

SSL context could not be created - error in cert or password is wrong
 HTTPServer - SSL will not be enabled

Not sure what this error is all about.

[richgalloway]

Make sure the new server has the correct SSL certificate(s) installed.

[sdkp03]

Thanks for the reply. I have been thinking on same lines and trying to find steps to install new certificates. However I did not find any steps to establish connection between DMC/LM and peers. Error message as extracted from our log file:

ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/ost_certs/model_lms_CA.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt

I have verified that the license file has no special characters. Have also verified that the existing cert can be accessed using below command:

openssl x509 -in /opt/splunk/etc/auth/ost_certs/model_lms_CA.pem -text -noout

Do you think following same steps as below would help me with the issue?

https://community.splunk.com/t5/Security/How-do-I-set-up-SSL-forwarding-with-new-self-signed-certifi...

[richgalloway]

I don't have enough experience with SSL to say for sure, but I think it's worth trying.

[isoutamo]

Have you check that your cert file contains the whole certificate chain?

Can you send your server.conf and outputs.conf?

r. Ismo

[sdkp03]

Eventually it has been found that the issue has been identified with the sslpassword. Old certificates were configured with sslpassword that started with a space. Example: " test123". Now when I try to store the same password in outputs.conf and restart the server, splunk is ignoring the leading space character. I have tried to add password in double quotes, single quotes, escaping with a forward slash - nothing seems to be helping with getting the right password encrypted. This is where am stuck now at. 

[isoutamo]

Hi

Can you try to change password of this certificate file? E.g. https://smallbusiness.chron.com/change-pass-phrase-ssl-certificates-29137.html

r. Ismo