How to upgrade the universal forwarder agents in linux from 7.3.0 to latest version.

[AviSharma8]

I need to upgrade the universal forwarder agents on the multiple instance from the current 7.3.0 to the latest version. Can we directly upgrade it or need to go step by step. Let me know the process with the best practice to upgrade it.

[splunkmarroko]

I believe it's do-able just make sure :
to go over few prerequisite things:

1 to check compatibility: refer to splunk documentation for more info about compatibility and update ways.

2- to check system requirements: ensure that your system meets the requiment for splunk 9.4 or whatever version you're upgrading to.
3- and most importantly DON'T forget to take a backup of your /etc in case something goes wrong.

[isoutamo]

Not only …/etc as UFs internal bookkeeping is under …/var you must backup also it.

[livehybrid]

Hi @AviSharma8 ,

Updated: According to the 8.2 docs its possible to upgrade a UF from 7.3->8.2 (Upgrading a universal forwarder directly to version 8.2 is supported from versions 7.3.x, 8.0.x, and...) and 9.4 supports an upgrade from 8.2 (Splunk supports a direct upgrade of a universal forwarder to version 9.4 from versions 8.2.x and hig...). This is contrary to the more broad upgrade path for a full Splunk installation (Not a UF) which has more steps due to more migrations taking place.

In the meantime - Its also worth mentioning that Splunk Enterprise version 9.0 and higher requires Linux kernel version 3.x or higher and has an updated OS support list -  Check supported OS at https://docs.splunk.com/Documentation/Splunk/9.4.1/Installation/Systemrequirements

I have seen other people upgrade 7.3 -> 9.x without issue in a single step, however your mileage may vary due to unknown migrations under the hood so you may wish to follow the official stance as above.

Here's the general process for upgrading using the .tgz package on Linux for each of your 7.3->8.2->9.4 steps:

1. Stop the existing Universal Forwarder:

   bash $SPLUNK_HOME/bin/splunk stop

2. (Optional, but Recommended) Back up configuration:

   bash cp -rp $SPLUNK_HOME/etc /tmp/splunk_etc_backup_$(date +%F)

3. Download and extract the new version:
   • Download the latest UF .tgz package for your Linux architecture from splunk.com.
   • Extract the package:

     bash tar xzf splunkforwarder*.tgz -C /opt

     Note: Ensure you extract it to the same parent directory as your existing $SPLUNK_HOME. For example, if your current UF is in /opt/splunkforwarder, extract the new version also into /opt.
4. Start the forwarder: This requires accepting the license on the first start after an upgrade.

$SPLUNK_HOME/bin/splunk start​ --accept-license --answer-yes

A few additional Tips:

1. Test the upgrade on a non-critical forwarder or a small group first.
2. For upgrading many forwarders, consider using deployment tools like Ansible, Puppet, Chef, or SaltStack, or the Splunk Deployment Server.
3. Always check the forwarder compatibility matrix to ensure the new UF version is compatible with your Splunk Enterprise indexers/deployment server.

Useful Documentation Links:

1. https://docs.splunk.com/Documentation/Forwarder/9.4.1/Forwarder/Upgradetheuniversalforwarder
2. https://lantern.splunk.com/Splunk_Platform/Product_Tips/Upgrades_and_Migration/Upgrading_the_Splunk_...

🌟Did this answer help you? If so, please consider:

• Adding kudos to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[isoutamo]

Actually You cannot update it directly from old to new unless it match those restrictions which are defined for for splunk servers too! Usually this means that you can jump over one version like 7.3.x -> 8.1.x -> 9.0.x -> 9.2.x -> 9.4.x. Also you must start UF on each steps for updating e.g. fishbucket DB and other things which has changed between versions and need some internal updates.

Of course you could remove old UF installation and install the newest versions from scratch into it. But then you need remember that this means:

• You lost your UF's GUID => you will get a new UF into sever point of view. Of course you can use same GUIF in .../etc/instance.cfg and keep old UF information in sever side
• splunk.secret will change which means that if you have any secrets/passwords in your old configurations and you try to use those you need to use plain versions and give UF crypt those again
• You lost information where your inputs are as you lost fishbucketdb which keep track of those => are are reingesting all files again which you have in this node

 Maybe something else which I forgot?

I know that updating from some version to another version could work without issues, but not for all. And those issues could arise later on, not immediately after you start a new version.

I also strongly recommend you to use OS's native sw packages instead of use tar versions. With this way it's much easier manage your OS level information as you could trust your package management sw information.

[livehybrid]

Hi @isoutamo 

I'm a little confused here as I was under the impression UFs were pretty stateless. They dont have Python, KVStore and do not locally index data? Compared to HF or other full Splunk Enterprise instances which definitely need to be updated to specific versions incrementally.

I've updated countless UFs From 7->9 without issue but happy to update my previous post if needed. 

Looking at the remote UF updater (https://docs.splunk.com/Documentation/Forwarder/1.0.0/ForwarderRemoteUpgradeLinux/Supporteduniversal...) This supports a minimum version 8.0.0 and upgrades directly to 9.x so I am content that this is feasible.

I know that for non-UF hosts there is a pretty strict upgrade path,  

[isoutamo]

Actually they said "it could update from 8.0 to 9.0+" but it didn't said "it could upgrade directly from 8.0 to 9.0+". And in https://docs.splunk.com/Documentation/Forwarder/1.0.0/ForwarderRemoteUpgradeLinux/Architecture they said "... validates the universal forwarder migration path from the current version to the destination version."

UF contains splunk some dbs (e.g. fishbucket in /opt/splunkforwarder/var/lib/splunk/). Time by time they change somehow (I don't know exactly how) the internals of db structure. Those changes must apply into those DBs when you are upgrading UFs. As I said there could be some cases where this is needed but I'm quite sure that updating from 7.3 -> 9.4 is not belonging to that sets.

[livehybrid]

Ah yes okay @isoutamo - That is a fair point. Whilst I've had success with this previously, there is no guarantee it will go the same way for @AviSharma8 ! 

The Remote updated app simply has the following check target_major_version <= current_major_version+1 and when I ran it was happy to do 8.0 -> 9.4!

Nevertheless, I will update my original post and point at the official stance on this. According to the 8.2 docs its possible to upgrade a UF from 7.3->8.2 (Upgrading a universal forwarder directly to version 8.2 is supported from versions 7.3.x, 8.0.x, and...) and 9.4 supports an upgrade from 8.2 (Splunk supports a direct upgrade of a universal forwarder to version 9.4 from versions 8.2.x and hig...)

In the meantime - Its also worth mentioning that Splunk Enterprise version 9.0 and higher requires Linux kernel version 3.x or higher and has an updated OS support list -  Check supported OS at https://docs.splunk.com/Documentation/Splunk/9.4.1/Installation/Systemrequirements

🌟 Did this answer help you? If so, please consider:

• Adding kudos to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

[isoutamo]

Earlier there was also restrictions that upside versions must be higher than downside, but this has removed on 9.x. This means that UF's versions cannot be higher than HF/Indexers and also HF's cannot be higher versions than Indexers.

I suppose that this was some kind of warranty for splunk to avoid some weird issues. I know that in many times those versions works w/o issues even UF has higher versions than IDXs have.