Silent upgrade from 8.3 to 9.02 Not working - pop up on mongod failed to start , KVstore?

[jcorcoran508]

msiexec.exe /qn /I splunkforwarder-9.0.2-17e00c557dc1-x64-release.msi DEPLOYMENT_SERVER="10.0.0.7:8089" SPLUNKUSERNAME=Admin SPLUNKPASSWORD=S@M3!! AGREETOLICENSE=Yes  LAUNCHSPLUNK=0

This appears to be an upgrade of Splunk.
--------------------------------------------------------------------------------)

Splunk has detected an older version of Splunk installed on this machine. To
finish upgrading to the new version, Splunk's installer will automatically
update and alter your current configuration files. Deprecated configuration
files will be renamed with a .deprecated extension.

You can choose to preview the changes that will be made to your configuration
files before proceeding with the migration and upgrade:

If you want to migrate and upgrade without previewing the changes that will be
made to your existing configuration files, choose 'y'.
If you want to see what changes will be made before you proceed with the
upgrade, choose 'n'.

Perform migration and upgrade without previewing configuration changes? [y/n] y

-- Migration information is being logged to 'c:\program files\splunkuniversalforwarder\var\log\splunk\migration.log.2022-12-31.15-42-09' --

Migrating to:
VERSION=9.0.2
BUILD=17e00c557dc1
PRODUCT=splunk
PLATFORM=Windows-AMD64

It seems that the Splunk default certificates are being used. If certificate validation is turned on using the default certificates (not-recommended), this may result in loss of communication in mixed-version Splunk environments after upgrade.

"c:\program files\splunkuniversalforwarder\etc\auth\ca.pem": already a renewed Splunk certificate: skipping renewal
"c:\program files\splunkuniversalforwarder\etc\auth\cacert.pem": already a renewed Splunk certificate: skipping renewal
Failed to start mongod.
Did not get EOF from mongod after 5 second(s).
[App Key Value Store migration] Starting migrate-kvstore.
Created version file path=c:\program files\splunkuniversalforwarder\var\run\splunk\kvstore_upgrade\versionFile36
[App Key Value Store migration] Collection data is not available.
ERROR - Failed opening "c:\program files\splunkuniversalforwarder\va

[dwest_splunk]

Hi @jcorcoran508 ,

Looks like you may be running into Known Issue for 9.0.2 on your UF

https://docs.splunk.com/Documentation/Splunk/9.0.2/ReleaseNotes/KnownIssues#Universal_forwarder_issu...

SPL-232467

SSL UF v9.x SSL Cert Auto-populates into Windows Certificate Store

Workaround:
Make the following changes in server.conf:
Before upgrading:

Save the original setting of [sslConfig] / serverCert if it's set, and then set the following:

[sslConfig] serverCert = C:\Program Files\SplunkUniversalForwarder\etc\auth\new_default.pem

[kvstore] disabled = 1

The file `C:\Program Files\SplunkUniversalForwarder\etc\auth\new_default.pem` must not exist before the upgrade.

After upgrading: Set serverCert back to the original value and restart the universal forwarder. If it was not set before applying the workaround, it can be left unchanged as the universal forwarder will use the newly generated PEM file.