Deploying and updating Splunkbase apps using a deployment server?

[rapplegate]

I'm running Splunk for Enterprise 7.3.0 on Ubuntu 18.04 doing a demo deployment with a sales trial license. It's a single instance deployment with only a handful of hosts, but the production deployment will separate out the roles to different servers.

I would like to deploy the Splunk App for Windows Infrastructure app and the other Windows add-ons to my Windows Universal Forwarders, as listed here: https://docs.splunk.com/Documentation/MSApp/1.5.2/MSInfra/HowtodeploytheSplunkAppforWindowsInfrastru... (not enough karma for links, sorry). It's to my understanding that I would have to do the following to prep an app for deployment:

1. Download the "Splunk Add-on for Windows" from Splunkbase (App 742) .tgz file.
2. Manually extract and copy the contents of the of the app to $SPLUNK_HOME/etc/deployment-apps/.
3. Manually have splunk rescan the directory with "splunk reload deploy-server", which is non-optional and not automatic.

This procedure is completely different then the easy GUI based approach when adding apps to my search head.

1. Click Apps -> Find More Apps
2. Search for the App through Splunkbase, even seeing which apps are already installed.
3. Click Install. Authenticate and accept the T&Cs.
4. Click Restart if needed.

If there's an update to an app installed via Splunkbase, and the app is visible, I can click the update button in the listed apps on the home page. To update the same deployed app on the same splunk instance, it appears I have to do the manual process.

Since my search head is also my deployment server, shouldn't installing deployable apps have the same ease and functionality? If I want to update a deployed app that's on Splunkbase, do I have to do this manual process for each Splunkbase app? Is there a GUI based way to install apps for deployment, be it either from Splunkbase or manually written? Am I missing something in my workflow? Is there an app that offers this functionality, or at least notifies me if a Splunkbase deployed app is out of date? I don't want to deploy outdated, broken, or exploitable apps, especially if there's a newer version available.

I can understand the need for maintaining older versions of deployed apps, and not wanting them to update when a Splunkbase maintainer updates their app, but I think there would be the option of at least updating the app through some process in the GUI, or at least notifying the user an update is available.

[gcusello]

Hi rapplegate,
the situation you described is ok only for a demo installation but in usual production architectures there are many target servers, this means that you cannot use a Search Head as Deployment Server and you have to use a dedicated one!
I agree with you that the deployment procedure is very intricate because you have to do some actions by CLI (the ones you listed) and some action by web that you didn't listed (create ServerClass and populate it with apps and targets).
I spoke about this with italian Splunk people that it's a very strange thing this way to proceed! but no changes until now, I continue to hope that someone shouts louder and someone else hears!

Anyway there is a way because you have separated environments between apps to display on SH and apps to deploy, with no automation of update: you have to maintain a strict control on deployed apps, otherwise you risk to lose control on your ingestion.

I hope to have been helpful for you.

Bye.
Giuseppe

[rapplegate]

Hi Giuseppe,
Thank you for your answer. I understand that in a production deployment there's usually many indexers and a separate search head as well as a separate deployment server, which is why I asked if this procedure would be different if I separated the server roles. For our production environment, we may be going completely with Splunk Cloud, with only a single Splunk instance on-prem acting as the deployment server.

I also purposely left out deploying the apps, since whatever apps are on the deployment server will automatically update the apps on the clients, so only updating the apps on the deployment server matters. Adding server classes, adding apps to server classes, and having the apps deployed works great, but it's the maintaining and installation of deployment apps on the deployment server is where I'm having trouble.

I understand the need for maintaining a strict control on the deployed apps for ingestion, but there's also the need for security, functionality, and maintenance. Is this just something that everyone does? Has someone introduced a workflow or app that makes this process smoother and easier? I've written a simple script that moves and extracts apps from a staging location, as well as tells Splunk to re-scan deployment apps as a workaround, but there's no way to ensure that in the future these apps are updated or deprecated. I'm not asking for automatic updates, even though that would be a nice option, but at least someway of knowing which ones were updated, just like on the search head.

You have been helpful, and for that I thank you. I just hope enough people see this to figure this out or point me in the right direction.

[gcusello]

The only way I know is a script as you described, but I suggest to think many times to this solution because upgrades usually must be customized: e.h. TA_Windows, by default, has all the inputs disabled and you must enable the ones you want, how do you manage this use case?
The best way is always manually manage the update process.

Bye.
Giuseppe

[rapplegate]

For updating an app, all customization should be in the local folder, which won't be touched with an update, correct? An update to an app will only update whatever the maintainer distributes, never any customizations. My simple script doesn't do anything that a user won't do normally, and it's to my understanding that every user is doing these same exact steps whenever they want to update or initially deploy a distributed app. For an initial install, the app won't be distributed until it's assigned to a class, which gives plenty of time to customize it the standard way. It would be nice if there was a GUI way to make those customizations, but that's outside of the scope of this question. I'm specifically asking about installing or updating an app on a distribution server, not deploying that app to the clients or the modifications that need to be done beforehand.

[gcusello]

Hi rapplegate,
yes customizations are in the local folder that isn't modified by upgrade, but in this way you lose control on your deployment.
For my experience, control is the first need of a deployment!
To have the max control I usually distribute TAs only with the default folder to be sure that older configurations are upgraded and there are only the configurations I want! in other words I don't use local folder to be sure of the active configurations.

Bye.
Giuseppe

[richgalloway]

Minor correction to the first #3: splunk reload deploy-server does not restart Splunk. It merely tells the DS to re-scan the deployment-apps directory.

There is no GUI for installing/updating apps in deployment-apps. If there was, it would be documented and you'd know about it.

[rapplegate]

I added your correction, thank you.

Thank you for answering. If there's no official documented way to update deployed apps via the Splunk GUI yet, is there a workflow, app, or add-on that provides this functionality, or at least makes it easier to check for updates? It seems kind of tedious and repetitive for it not to be automated and/or simplified, especially if there's a large number of apps that need to be maintained. Is a Splunk admin going to manually check each app and add-on they've deployed from Splunkbase and check to see if it matches the version they're running? The only other option I can see is subscribing to each add-on deployed on Splunkbase, to at least get notifications that it's been updated.

[richgalloway]

I'm not aware of an app or add-on that will compare deployment server apps against Splunkbase. It's a good idea, though.