What are best practices for how to deploy an add-on such that different servers run the same add-on, but with slightly different configurations? A common situation is when the add-on has different inputs enabled or disabled depending on the host.
I have additional constraints. I want to continue to use a single deployment server, not multiple. All of the add-ons I'm using are downloaded from Splunkbase, so I am sensitive to changes that could complicate upgrades of the add-on as they are published. Lastly, some of the add-ons have scripted inputs for which I know there are considerations regarding the script path.
For example, I want to do this for Splunk Add-on for Unix and Linux. Some inputs are going to be enabled everywhere, like those for CPU, disk, and memory. Others will be enabled on only some deployment clients based on the needs of the client machine's owners. It's clear from the documentation how to deploy this add-on with some inputs enabled within the configuration file
local/inputs.conf. But what about the scripted inputs that need to be enabled on other clients? And what about deploying apps to the deployment server itself?
I know there are many variables, and thus many possible solutions. So rather than focus on a single 'best' practice, I'd love for everyone to share any approaches you might recommend, why, and what risks or "gotchas" to be aware of. I'll collect and curate all of your ideas into a single answer for future readers, and reward you with karma points.
So have at it! What practices have you used and why do you consider them 'best'?
I'll get the ball rolling with a few approaches I've considered, what works about them and where they get risky. But first, let's review some basics of configuration management that are critical for understanding this conversation.
The Splunk Enterprise Admin Manual provides excellent coverage About configuration files. Here's some critical topics within that section and how they relate to this discussion.
Here are some additional topics in other manuals that are important to know about.
btoolshows what the blend of configuration looks like if Splunk reloads the configuration.
Here are some options for deploying multiple copies of an add-on with different configurations, where the options work well, the complexity or level of effort to implement, and where the options run into challenges.
localadditions. Only deploy the clones, not the original app, since it will be redundant and unused.
Splunk_TA_windowsfrom Splunkbase for my AWS environment, I'll name the variation
Splunk_TA_windows-aws, which identifies it as a child of the parent app
localinto a new app. In the new app, populate the
localfolder with the desired configuration for the target clients. The new app will now contain all of the substance of the original app, but with your
localadditions. Deploy only the new app, not the original app since it will be redundant and unused.
Can you think of others? Anything else worth adding or considerations of these approaches?
Update: changed recommendation of Parent and children approach to discourage it's use when scripted inputs involved. In practice, the scripted inputs require hard coded fully qualified paths and hardcoded source and sourcetypes and indexes for this to work. In total, this amount of change renders the Parent and Child solution as a bad practice when scripted inputs are involved.