Universal Forwarder Central Upgrade Server

[verbal_666]

Hi all.

I really did never understand WHY Splunk Developments never take in mind to develop a Central Upgrader/Downgrader Server to manage UFs. Since it's a simple operation to upgrade or downgrade an agent, WHY do not introduce a Splunk Instance to make it???

I actually do it by special addons and scripts with special OS tricks commands which launch the remote procedure by UF scheduler itsself directly and then autorestart itsself (both Win and Linux) in a medium/great environment (about 2000 UFs). I can't manage it by Dev-OPS tools, so i developed a special UF addon to do it.

BUT, why not make SPLUNK itsself do it??? 🤔🤔🤔🙄🙄🙄let's suppose to introduce a module in Deployer or Deployment Instance that take all UF versions and a button to update/downgrade to a new/old version?

[verbal_666]

Thread can be closed.

Thanks 👍👍👍

[PickleRick]

1. There have been some more or less unofficial ways of upgrading the forwarders for ages. If I remember correctly, Splunk 10 introduced some limited UF upgrade functionality

2. It's not as easy to do as said. First and foremost - Splunk very often runs with a minimal required privilege account which means it cannot perform administrative tasks on the OS. Which in turn means that while you maybe could overwrite Splunk's own files (but still with a hardened installation you might have a hard time doing even that), you can't touch system's packages database. So if you have - for example - a UF installed from a RPM package, overwriting its files manually would cause mismatch vs. RPM database of installed files.

So there is much more to the upgrade process than meets the eye.

[verbal_666]

1. as said, i developed my own "Upgrader Addon", and works, both on Windows and Linux... but to restart the new UF there's lot of tricks to make it happens in detached mode by original UF scheduler which stop the restart if you try a normal restart!!! 🙄good to know in V10 there's something about 👍👍👍

2. OK, but let the final user decide if use the feature or not!!! So, in that case, i run UF with root. We have safe perimeters, and we need root to read some special files in many ways, so we decided to use root as normal user. Another way should be a separate upgrader process with root privilege to make ONLY a stop/copy/restart new/old binaries (tgz) of UFs. There are many many ways to make it and make it easy, not discarding security!

IMO, SPLUNK needs an official UF Upgrader Instance/Procedure.

[PickleRick]

You're looking at this throught your specific use case and your particular needs. And your specific case is very unusuall and generally discouraged. Running the forwarder as root is a Bad Idea(tm).

Contrary to what some people believe, the forwarder as such is not a security tool, it's a data gathering tool. True, the data gathered with a forwarder can be used for security, but can be used for plethora of other things. And as such - since it's just a "probe" for getting data, it's limited in what it can do and how it does it.

Running the forwarder as root (and allowing "external" people manage it) introduces several relatively heavy risks. And since the tool was developed for a completely different purpose that - let's say - an endpoint security agent, it has different controls implemented. And while a security team typically manages your endpoint agents, their actions are usually heavily monitored and audited and there usually several limitations as to what the agent can do, there are different roles with different privileges and so on. With Splunk running as root, "anyone" can run anything on your forwarder box and not leave much traces behind.

So Splunk should not be run as root. Period.

There are other possibilities to automate upgrade but each has its own set of challenges. And in order to ship something with your product you should have a reliable solution (yes, I know, I'm a bit idealistic here), not just some duct tape and zip tied set of scripts which will fail on a first possible opportunity.

[verbal_666]

In a secured, controlled, monitored, firewalled environment, such as login into systems with any own user id/token/traced_local_ip, and where also the remote shell logins are recorded (with security professional tools), the risk is very poor. Running any process is root user/group is dangerous, but depends always by the general environment.

PS. as said, in Windows you install/run UF by default with LOCAL\SYSTEM...

[PickleRick]

In Windows you no longer install by default with Local System, as far as I remember. (and yes, it used to be my private pet peeve).

BTW, the principle of least privilege, does it ring a bell? 

And please stop spamming the thread with shouting that you want to run your forwarder as root and want it to upgrade by itself.

You got an answer why it's more complicated than your single unsupported use case. What else do you expect? Engineering rushing in to apologize for not having implemented this functionality yet and making a commitment to deliver it within two weeks? Sorry, not gonna happen.

If you think a feature is desired, you can vote for it on ideas.splunk.com (or post your own one if you don't find an existing one). That's it, end of story.

[verbal_666]

Here it is,

[PickleRick]

That is interesting. Are you sure it's not an old version? The docs say that "By default the universal forwarder is installed with a least-privileged user. You can use the radio buttons to change the account on which the universal forwarder runs."

https://help.splunk.com/en/splunk-enterprise/forward-and-process-data/universal-forwarder-manual/10....

I don't usually deal with manual installation of windows forwarder so if there is some discrepancy between the docs and the actual installer's behaviour I wouldn't know it. It would be worth raising a support case if there was.

[verbal_666]

UF 9.4.4 ... you can try yourself 😎

[PickleRick]

Interesting 🙂 The docs for 9.4 also say that it's installed with least priv user. Worth testing and raising a case.

[livehybrid]

Hi @verbal_666 

As @PickleRick mentioned, there is a feature in Splunk 10 already for this. Please check out https://help.splunk.com/en/splunk-enterprise/forward-and-process-data/splunk-remote-upgrader-for-lin... which gives some information on how to get started. 

This will then allow you to remotely upgrade your UF as required. 

I understand you are running as root, which is ultimately your decision but is *strongly* discouraged. There are many ways (e.g setfacl) to allow a non-root user access specific files which are much safer than running as root. If you are running a deployment server and running the UFs that connect to it as root then a bad actor (or even accidental bad deployment) with access to the DS could take potentially take control of all the UF and carry out commands as the root user. 

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[verbal_666]

Just for your information... under Windows, UF starts and runs by default with LOCAL\SYSTEM special user.

[tscroggins]

If you're a Windows shop large enough to warrant centralized software management, you're likely already using Configuration Manager or Intune or even group policy to push software.