How to troubleshoot why 1 indexer in a Splunk indexer cluster crashed and won't restart with a "Bad Decrypt" error?

[basher590]

HI,

I have inherited a clustered Splunk setup and I noticed that 1 of my 2 indexers had crashed a couple of days ago.
Trying to restart it yields a Splunk timed out waiting to start error. Looking at the splunkd log I see the following error:

02-22-2016 14:05:35.800 +0000 ERROR SSLCommon - Can't read key file C:\Program Files\Splunk\etc\auth\server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.

The key file is there and looks OK to me, though I am not sure how I can test it. I did use the OpenSSL command, but received the same message. I tried changing the password in the config file and I receive a "bad password" error, so I know the PW is correct and it is reading the correct file.

There have been no updates or config changes that I am aware of, this 1 indexer server just seemed to crash.

Is it just a case of creating a new certificate on this one indexer, or are there other steps that need to be followed so I don't break the cluster / indexes?

I am running
Splunk Version
6.2.3
Splunk Build
264376

On Windows 2012 R2 servers.

Thanks

[basher590 · Accepted solution]

I got this fixed in the end by creating a new certificate and applying it to the faulty server.
The first restart worked but I received a new error relating to http://127.0.0.1 instead of https, but after another restart it cleared and all was good.