topic Re: Server certificate renewal failed in Splunk Enterprise

Server certificate renewal failed

gc — Fri, 03 May 2024 13:55:46 GMT

Hello there,

I have a problem with one of our Splunk installations on Windows. The server certificate is expired and I'm unable to renew it. I've tried renaming C:\Program Files\Splunk\etc\auth\server.pem and restarting Splunk, which ends with that:

The certificate generation script did not generate the expected certificate file:C:\Program Files\Splunk\etc\auth\server.pem. Splunkd port communication will not work.
SSL certificate generation failed.

And I also tried this command: C:\Program Files\Splunk\bin>splunk createssl server-cert -d "C:\Program Files\Splunk\etc\auth" -n server -c *servername*
Which also fails with this:

CreateProcess: error 193
Command failed (ret=-1), exiting.

Anyone knows how to fix this? Thanks in advance.
Best regards

Alex

Re: Server certificate renewal failed

deepakc — Fri, 03 May 2024 16:33:21 GMT

1. Check Your Admin Permissions etc

2. Could it be AV / blocking the action - command?

Re: Server certificate renewal failed

gc — Wed, 08 May 2024 08:26:38 GMT

Hello,

thanks for replying, checked the permission and disabled AV, still the same outcome. Any other ideas?

Best regards
Alex

Re: Server certificate renewal failed

gc — Fri, 17 May 2024 10:33:06 GMT

Hi,

anyone else with a suggestion? 😕
Thanks again, best regards

Alex

Re: Server certificate renewal failed

deepakc — Fri, 17 May 2024 11:07:56 GMT

Try this, not sure if it will work, but worth a try.

See if the variable is pointing to this file which contains SSL config / library's etc

echo %OPENSSL_CONF%

Set it as below and try again.

set OPENSSL_CONF=c:\Program Files\Splunk\openssl.cnf

Re: Server certificate renewal failed

gc — Fri, 17 May 2024 13:08:21 GMT

Hi there,

thank you for your idea, but unfortunately it was not working:

The path is correct. Is there any way to find out, why the generation is failing? Checked some logs, but couldn't find anything that was helping...

Re: Server certificate renewal failed

deepakc — Fri, 17 May 2024 13:39:01 GMT

There may be something in splunkd.log(not sure) find this in $SPLUNK_HOME\var\log\splunk

Whats the output of this? (I'm starting to think the root cacert.pem has something to do with this.)

openssl x509 -in "c:\Program Files\Splunk\etc\auth\cacert.pem" -text -noout

Does it show its expired? may be this has something to do with it.

Try and rename that file cacert.pem or it could be ca.pem and do a restart

Re: Server certificate renewal failed

gc — Fri, 17 May 2024 15:01:22 GMT

I have checked the log, there is nothing there. In fact there is only 1 log with new entries. These are the last entries from splunkd-utility.log:

05-17-2024 16:44:40.570 +0200 INFO ServerConfig - Found no hostname options in server.conf. Will attempt to use default for now.
05-17-2024 16:44:40.570 +0200 INFO ServerConfig - Host name option is "".
05-17-2024 16:44:40.570 +0200 INFO ServerConfig - TLS Sidecar disabled
05-17-2024 16:44:40.570 +0200 WARN SSLOptions - server.conf/[sslConfig]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security
05-17-2024 16:44:40.570 +0200 INFO ServerConfig - No 'C:\Program Files\Splunk\etc\auth\server.pem' certificate found. Splunkd communication will not work without this. If this is a fresh installation, this should be OK.
05-17-2024 16:44:40.586 +0200 INFO ServerConfig - disableSSLShutdown=0
05-17-2024 16:44:40.586 +0200 INFO ServerConfig - Setting search process to have long life span: enable_search_process_long_lifespan=1
05-17-2024 16:44:40.586 +0200 INFO ServerConfig - enableTeleportSupervisor=0, scsEvironment=production
05-17-2024 16:44:40.586 +0200 INFO ServerConfig - certificateStatusValidationMethod is not set, defaulting to none.
05-17-2024 16:44:40.586 +0200 INFO ServerConfig - Splunk is starting with EC-SSC disabled

cacert.pem is valid till 2027 and I have checked server.conf, which has no entry for hostname. But this seems to be normal, have checked against another installation.

Re: Server certificate renewal failed

deepakc — Fri, 17 May 2024 15:23:19 GMT

That WARN is just for extra security.

Its still having issues with the server.pem file

I'm out of options to check mate, consider logging a support call, or you could if this is an option to you, backup /etc/apps folder and re-install Splunk, and restore the backed up /etc/apps folder, I know this is a drastic step...but might be quicker.

Re: Server certificate renewal failed

gc — Tue, 21 May 2024 14:44:38 GMT

Thanks, I did a reinstallation. And just to be sure, I had to save + restore \var\lib\splunk
Thanks for your help, have a good day 🙂