Splunk Universal Forwarder Upgrade From 8.2.4 to 9.0.2

[anandhalagaras1]

Hi All,

We are running with Splunk UF version 8.2.4 in our Linux x64 client machines and we have planned to get them upgraded to the latest version 9.0.2 hence i have downloaded the latest rpm package and usually we used to deploy the package using rpm in all our client servers so when we tried to deploy the package using RPM we are getting the below error.

So do we have anything needs to be done from our end before performing the upgrade of Splunk UF on Windows and Linux servers from 8.x to 9.x version?

"/opt/splunkforwarder/etc/auth/ca.pem": already a renewed Splunk certificate: skipping renewal
"/opt/splunkforwarder/etc/auth/cacert.pem": already a renewed Splunk certificate: skipping renewal
Failed to start mongod.
Did not get EOF from mongod after 5 second(s).
[App Key Value Store migration] Starting migrate-kvstore.
Created version file path=/opt/splunkforwarder/var/run/splunk/kvstore_upgrade/versionFile36
Created version file path=/opt/splunkforwarder/var/run/splunk/kvstore_upgrade/versionFile40
[App Key Value Store migration] Collection data is not available.
Starting KV Store storage engine upgrade:
Phase 1 (dump) of 2:
Failed to migrate to storage engine wiredTiger, reason=Failed to receive response from kvstore error=, service not ready after waiting for timeout=300271ms
[App Key Value Store migration] Starting migrate-kvstore.
Created version file path=/opt/splunkforwarder/var/run/splunk/kvstore_upgrade/versionFile42
[App Key Value Store migration] Collection data is not available.
[DFS] Performing migration.
[DFS] Finished migration.
[Peer-apps] Performing migration.
[Peer-apps] Finished migration.
Creating unit file...
Current splunk is running as non root, which cannot operate systemd unit files.
Please create it manually by 'sudo splunk enable boot-start' later.
Failed to create the unit file. Please do it manually later.

Systemd unit file installed by user at /etc/systemd/system/SplunkForwarder.service.
Configured as systemd managed service.

Nov 09 07:05:49 splunk[135425]: Invalid key in stanza [webhook] in /opt/splunkforwarder/etc/system/default/alert_actions.conf, line 229: enable_allowlist (value: false).
Nov 09 07:05:49 splunk[135425]: Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
Nov 09 07:05:49 splunk[135425]: Checking conf files for problems...
Nov 09 07:05:49 splunk[135425]: Done
Nov 09 07:05:49 splunk[135425]: Checking default conf files for edits...
Nov 09 07:05:49 splunk[135425]: Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.0.2-17e00c557dc1-linux-2.6-x86_64-manifest'
Nov 09 07:05:49 splunk[135425]: PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security
Nov 09 07:05:49 splunk[135425]: 2022-11-09 07:05:49.925 -0600 splunkd started (build 17e00c557dc1) pid=135425

[richgalloway · Accepted solution]

If the UF is running then the messages can be ignored.  If the UF is not running then contact Splunk support.

Universal Forwarders do not use KVStore so all messages related to mongodb (including wiredTiger) can be ignored if the UF is running. 

Since this is an upgrade, a systemd unit file should already be present so you can ignore the messages about that.

The webhook warning can be ignored, as well.  It does not apply to UFs.  Report that to Splunk, too.

[anandhalagaras1]

@richgalloway Thank your for your response.

Anyhow I have raised a case with Support regarding the errors post UF upgrade to the latest version.

[cweckel2000]

I'm running into the same issue with my upgrade from 8.2.x to 9.0.2.  The upgrade seems to somewhat work as Splunk Cloud reports the UF as 9.0.2 and logs are ingestion; however, locally the UF still shows as v8.2.x after the upgrade.

Can you share what Splunk support comes back with?  I may also open a ticket. 

[anandhalagaras1]

@cweckel2000 , Splunk Support stated to ignore the error and perform the upgrade as mentioned by @richgalloway  Since it seems to be a bug and they have confirmed that they have internally raised a JIRA ticket with their internal Development team regarding the issue and it will be sorted out in the future release.

[cweckel2000]

Awesome, thanks for confirming that!  FWIW, I found the following sequence to be the most reliable when upgrading our UFs:

1. Disable the KVStore in server.conf (this gets rid of a ~15min timeout during the upgrade)

2. Start the upgrade

3. Stop the Splunk service

2 and 3 might seem backwards but I'm deploying the upgrade as a TA from the deployment server using powershell so if I stop the service first, it'll end my script.