We did rebuild existing server that hosted LM and DMC. I did install latest splunk on the rebuilt server. Copied configuration settings from old build. Now when I try to turn on splunk service, the service does start up fine but when I try accessing it from UI, I see below mentioned errors logged in splunkd.log:
Error in Indexer Discovery communication. Verify
that the pass4SymmKey set under [indexer_discovery:dr_indexer_cluster_group] in 'outputs.conf' matches the same setting
under [indexer_discovery] in 'server.conf' on the Cluster Master. [uri=https://splunk-idx-cm.trgtm.ostravam.corp.tels
tra.com:8089/services/indexer_discovery http_code=502 http_response="Error connecting: SSL not configured on client"]
I did decrypt and verified that the pass4symmkey is same on CM and the new server.
> SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 err
or:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt
SSL context could not be created - error in cert or password is wrong
HTTPServer - SSL will not be enabled
Not sure what this error is all about.
Thanks for the reply. I have been thinking on same lines and trying to find steps to install new certificates. However I did not find any steps to establish connection between DMC/LM and peers. Error message as extracted from our log file:
ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/ost_certs/model_lms_CA.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt
I have verified that the license file has no special characters. Have also verified that the existing cert can be accessed using below command:
openssl x509 -in /opt/splunk/etc/auth/ost_certs/model_lms_CA.pem -text -noout
Do you think following same steps as below would help me with the issue?
Eventually it has been found that the issue has been identified with the sslpassword. Old certificates were configured with sslpassword that started with a space. Example: " test123". Now when I try to store the same password in outputs.conf and restart the server, splunk is ignoring the leading space character. I have tried to add password in double quotes, single quotes, escaping with a forward slash - nothing seems to be helping with getting the right password encrypted. This is where am stuck now at.