Splunk Remote Upgrader for Linux Universal Forwarders from Deployment Server

[msmadhu]

We are attempting to upgrade Splunk Universal Forwarders using the UF Remote Upgrade Add-on.

As per Splunk documentation, we have installed the add-on, placed the required upgrade packages in the appropriate directories and pushed the app to the target clients via the Deployment Server. However, the upgrade is not taking place and we are seeing the following error in the log file located at /opt/splunk/var/log/splunk/upgrader_package_delivery.log:

Error log from  /opt/splunk/var/log/splunk/upgrader_package_delivery.log

2025-09-22-14:22:10 Conf file from UF updater does not exist at "/opt/splunk/var/run/splunk/splunkupdater/info". The UF updater is likely not installed or running.

2025-09-22-14:22:10 Cancelling package delivery and waiting for next interval.

2025-09-22-14:23:10 Checking if any forwarder packages are available

2025-09-22-14:23:10 Found files in /opt/splunk/etc/deployment_apps/splunk_app_uf_remote_upgrade_linux/bin/../local/packages. Will deliver them.

splunk_app_uf_remote_upgrade_linux/
├── bin
│   └── SPLUNK_UPDATER_INTERNAL_deliver_pkg.sh
├── default
│   ├── app.conf
│   ├── inputs.conf
│   ├── local_config
│   └── packages
│   ├── splunk-upgrader-linux-102.tgz
│   └── splunk-upgrader-linux-102.tgz.sig
├── local
│   ├── app.conf
│   ├── local_config.bkp
│   └── packages
│   ├── splunkforwarder-10.0.0-e8eb0c4654f8-linux-amd64.tgz
│   ├── splunkforwarder-10.0.0-e8eb0c4654f8-linux-amd64.tgz.sha512
│   └── splunkforwarder-10.0.0-e8eb0c4654f8-linux-amd64.tgz.sig
├── metadata
│   └── local.meta
└── VERSION

Note: Splunk enterprise is running with version 10.0.0 and UF is running with 9.4.2

[msmadhu]

Hi @livehybrid 

Could u plz help on the fix,

splunk_updater_launcher.sh[2216934]: /opt/splunkupgrader/bin/splunk_updater_launcher.sh: line 14: SPLUNK_HOME: unbound variable
systemd[1]: splunk-upgrader.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: splunk-upgrader.service: Failed with result 'exit-code'.
systemd[1]: splunk-upgrader.service: Service RestartSec=100ms expired, scheduling restart.
systemd[1]: splunk-upgrader.service: Scheduled restart job, restart counter is at 5.
systemd[1]: Stopped Splunk Upgrader daemon that monitors for new UF packages to upgrade to.
systemd[1]: splunk-upgrader.service: Start request repeated too quickly.
systemd[1]: splunk-upgrader.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Splunk Upgrader daemon that monitors for new UF packages to upgrade to.

[msmadhu]

tail -15f upgrade.log
2025-12-01-11:11:00 INFO Checking insensitive config: MONITOR_PKG_INTERVAL_SEC=
2025-12-01-11:11:00 INFO Checking insensitive config: FWD_UPGRADE_TIMEOUT_SEC=
2025-12-01-11:11:00 INFO Checking insensitive config: FWD_UPGRADE_MAX_RETRY=
2025-12-01-11:11:00 INFO Checking insensitive config: ROTATE_HISTORY_LOG_DAYS=
2025-12-01-11:11:00 INFO Checking sensitive config: SPLUNK_UPDATER_USER=
2025-12-01-11:11:00 INFO Checking sensitive config: SPLUNK_UPDATER_GROUP=
2025-12-01-11:11:01 INFO Checking insensitive config: SPLUNK_HOME=/opt/splunkforwarder
2025-12-01-11:11:01 INFO Validating config SPLUNK_HOME=/opt/splunkforwarder

[msmadhu]

Hi @livehybrid 

Pushed UF upgrade files from Deployment server to UF and same files are reflecting in UF, but not upgraded

No, service is not running,

systemctl status splunk-upgrader.service
Unit splunk-upgrader.service could not be found.

UF:

[root@xyz apps]# tree splunk_app_uf_remote_upgrade_linux/
splunk_app_uf_remote_upgrade_linux/
├── bin
│   ├── pkg_delivered
│   └── SPLUNK_UPDATER_INTERNAL_deliver_pkg.sh
├── default
│   ├── app.conf
│   ├── inputs.conf
│   ├── local_config
│   └── packages
│   ├── splunk-upgrader-linux-102.tgz
│   └── splunk-upgrader-linux-102.tgz.sig
├── local
│   ├── app.conf
│   └── packages
│   ├── checksums.txt
│   ├── splunkforwarder-10.0.0-e8eb0c4654f8-linux-amd64.tgz
│   ├── splunkforwarder-10.0.0-e8eb0c4654f8-linux-amd64.tgz.sha512
│   └── splunkforwarder-10.0.0-e8eb0c4654f8-linux-amd64.tgz.sig
├── metadata
│   └── local.meta
└── VERSION

6 directories, 14 files

/opt/splunkforwarder/bin/splunk version
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk:splunk /opt/splunkforwarder"
Splunk Universal Forwarder 9.4.2 (build e9664af3d956)

[livehybrid]

Hi @msmadhu 

Can you confirm you have configured the remote update service on the remote UF as per the docs at https://help.splunk.com/en/splunk-enterprise/forward-and-process-data/splunk-remote-upgrader-for-lin...

Please could you post the output of: systemctl status splunk-upgrader

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful

• Marking it as the solution if it resolved your issue

• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

[msmadhu]

Hi @livehybrid 

● splunk-upgrader.service - Splunk Upgrader daemon that monitors for new UF packages to upgrade to
Loaded: loaded (/etc/systemd/system/splunk-upgrader.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2025-12-01 11:11:01 UTC; 40s ago
Process: 2011130 ExecStart=/opt/splunkupgrader/bin/splunk_updater_launcher.sh (code=exited, status=1/FAILURE)
Main PID: 2011130 (code=exited, status=1/FAILURE)

Dec 01 11:11:01 xyz systemd[1]: splunk-upgrader.service: Service RestartSec=100ms expired, scheduling restart.
Dec 01 11:11:01 xyz systemd[1]: splunk-upgrader.service: Scheduled restart job, restart counter is at 5.
Dec 01 11:11:01 xyz systemd[1]: Stopped Splunk Upgrader daemon that monitors for new UF packages to upgrade to.
Dec 01 11:11:01 xyz systemd[1]: splunk-upgrader.service: Start request repeated too quickly.
Dec 01 11:11:01 xyz systemd[1]: splunk-upgrader.service: Failed with result 'exit-code'.
Dec 01 11:11:01 xyz systemd[1]: Failed to start Splunk Upgrader daemon that monitors for new UF packages to upgrade to.

drwxrwxrwx. 7 splunkupgrader splunkupgrader 88 May 23 2025 splunkupgrader

2025-12-01-11:14:57 Found files in /opt/splunkforwarder/etc/apps/splunk_app_uf_remote_upgrade_linux/bin/../local/packages. Will deliver them.
2025-12-01-11:14:57 Conf file from UF updater does not exist at "/opt/splunkupdater/info". The UF updater is likely not installed or running.
2025-12-01-11:14:57 Cancelling package delivery and waiting for next interval.