All Apps and Add-ons

What are best practices for deploying an add-on with slightly different configurations?

sloshburch
Splunk Employee
Splunk Employee

What are best practices for how to deploy an add-on such that different servers run the same add-on, but with slightly different configurations? A common situation is when the add-on has different inputs enabled or disabled depending on the host.

I have additional constraints. I want to continue to use a single deployment server, not multiple. All of the add-ons I'm using are downloaded from Splunkbase, so I am sensitive to changes that could complicate upgrades of the add-on as they are published. Lastly, some of the add-ons have scripted inputs for which I know there are considerations regarding the script path.

For example, I want to do this for Splunk Add-on for Unix and Linux. Some inputs are going to be enabled everywhere, like those for CPU, disk, and memory. Others will be enabled on only some deployment clients based on the needs of the client machine's owners. It's clear from the documentation how to deploy this add-on with some inputs enabled within the configuration file local/inputs.conf. But what about the scripted inputs that need to be enabled on other clients? And what about deploying apps to the deployment server itself?

I know there are many variables, and thus many possible solutions. So rather than focus on a single 'best' practice, I'd love for everyone to share any approaches you might recommend, why, and what risks or "gotchas" to be aware of. I'll collect and curate all of your ideas into a single answer for future readers, and reward you with karma points.

So have at it! What practices have you used and why do you consider them 'best'?

1 Solution

sloshburch
Splunk Employee
Splunk Employee

The Splunk Product Best Practices team provided this response. Read more about How Crowdsourcing is Shaping the Future of Splunk Best Practices.

I'll get the ball rolling with a few approaches I've considered, what works about them and where they get risky. But first, let's review some basics of configuration management that are critical for understanding this conversation.

Configuration management topics in Splunk Docs

The Splunk Enterprise Admin Manual provides excellent coverage About configuration files. Here's some critical topics within that section and how they relate to this discussion.

Here are some additional topics in other manuals that are important to know about.

  • Use btool to troubleshoot configurations - btool shows what the blend of configuration looks like if Splunk reloads the configuration.
  • Updating Splunk Enterprise Instances - this manual describes how the deployment server and deployment clients work to manage updates to Splunk Enterprise instances.
  • Where to place the scripts for scripted inputs - this topic describes how to manage scripted inputs, and how to ensure that relative paths are calculated correctly when Splunk Enterprise computes the fully qualified path to the script.
  • About installing Splunk add-ons - This scenario focuses on special considerations for distributed deployments and Splunk Cloud topologies.
  • Watch the following video about the suggested naming convention for Splunk apps and add-ons.

Naming apps. A suggested naming convention.

Potential solutions for deploying multiple copies of an add-on with different configurations

Here are some options for deploying multiple copies of an add-on with different configurations, where the options work well, the complexity or level of effort to implement, and where the options run into challenges.

Brute force

  • Approach: Deploy the add-on with a configuration that is common to all clients, then manually add configurations to the individual clients that need deviations.
  • Level of effort: Trivial
  • Gotchas: Hard to find and manage deviations at scale. Requires adjustments to accommodate the relative paths for scripted inputs.

Clones

  • Approach: Completely duplicate the app into a another app. Customize each clone to the specific needs of its target client. Deploy each specific clone to its target client. Each clone now contains all of the substance of the original app, but with your local additions. Only deploy the clones, not the original app, since it will be redundant and unused.
  • Level of effort: Implementation is moderate. Since upgrading each clone requires duplicating all over again, such recurring maintenance may be undesirable amount of work. Scripted inputs, relatively defined, will work natively.
  • Gotchas: Upgrades can be error prone and labor intensive, since you effectively recreate each clone. You may also need a naming convention to identify the source and its various clones. Hard-coded references to the original app name will cause errors, since the original app is not deployed.

Parent and children

  • Approach: Deploy the app with a configuration that is common to all clients and call it the 'parent'. Create 'children' that contain only the deviations needed for groups of clients. Deploy the children to their targets.
  • Level of effort: Moderate to large. Since all configuration is centralized, this approach scales well to clients, but becomes difficult to corral as the number of children increases (get it?).
  • Gotchas: Sharing scripted inputs in this model causes many other changes in order to work and therefore is not recommended. Also requires a naming convention to identify all the children and associate them with their parents. For example, when creating a variation of the Splunk_TA_windows from Splunkbase for my AWS environment, I'll name the variation Splunk_TA_windows-aws, which identifies it as a child of the parent app Splunk_TA_windows.

Symbolic links

  • Approach: Create symbolic links (symlinks) of all folders except local into a new app. In the new app, populate the local folder with the desired configuration for the target clients. The new app will now contain all of the substance of the original app, but with your local additions. Deploy only the new app, not the original app since it will be redundant and unused.
  • Level of effort: Implementation is complex. Upgrading is trivial.
  • Gotchas: There are passionate debates about symlinks because of the obfuscation introduced by the symlink abstraction. You may need a naming convention to identify the source and its various clones. Hard-coded references to the original app name will cause errors, since the original app is not deployed.

More?

Can you think of others? Anything else worth adding or considerations of these approaches?

View solution in original post

splunkreal
Motivator

Hello,

So Bruteforce or Clone is most recommended to use same bin scripts accross different apps for different hosts?

Thanks.

* If this helps, please upvote or accept solution 🙂 *
0 Karma

isoutamo
SplunkTrust
SplunkTrust
Hi
It's hard to say that any of those are "recommended". To be honest you must count pros and cons based on your need and your environment. It depends also which TA is in use and which options are really available for you. E.g. some TA:s are using own HEC instance and changing this is really hard or almost impossible (in practically).
r. Ismo
0 Karma

splunkreal
Motivator

Brute force looks good with modification of the path of scripted inputs in the case of Splunk_TA_nix

* If this helps, please upvote or accept solution 🙂 *
0 Karma

sloshburch
Splunk Employee
Splunk Employee

The Splunk Product Best Practices team provided this response. Read more about How Crowdsourcing is Shaping the Future of Splunk Best Practices.

I'll get the ball rolling with a few approaches I've considered, what works about them and where they get risky. But first, let's review some basics of configuration management that are critical for understanding this conversation.

Configuration management topics in Splunk Docs

The Splunk Enterprise Admin Manual provides excellent coverage About configuration files. Here's some critical topics within that section and how they relate to this discussion.

Here are some additional topics in other manuals that are important to know about.

  • Use btool to troubleshoot configurations - btool shows what the blend of configuration looks like if Splunk reloads the configuration.
  • Updating Splunk Enterprise Instances - this manual describes how the deployment server and deployment clients work to manage updates to Splunk Enterprise instances.
  • Where to place the scripts for scripted inputs - this topic describes how to manage scripted inputs, and how to ensure that relative paths are calculated correctly when Splunk Enterprise computes the fully qualified path to the script.
  • About installing Splunk add-ons - This scenario focuses on special considerations for distributed deployments and Splunk Cloud topologies.
  • Watch the following video about the suggested naming convention for Splunk apps and add-ons.

Naming apps. A suggested naming convention.

Potential solutions for deploying multiple copies of an add-on with different configurations

Here are some options for deploying multiple copies of an add-on with different configurations, where the options work well, the complexity or level of effort to implement, and where the options run into challenges.

Brute force

  • Approach: Deploy the add-on with a configuration that is common to all clients, then manually add configurations to the individual clients that need deviations.
  • Level of effort: Trivial
  • Gotchas: Hard to find and manage deviations at scale. Requires adjustments to accommodate the relative paths for scripted inputs.

Clones

  • Approach: Completely duplicate the app into a another app. Customize each clone to the specific needs of its target client. Deploy each specific clone to its target client. Each clone now contains all of the substance of the original app, but with your local additions. Only deploy the clones, not the original app, since it will be redundant and unused.
  • Level of effort: Implementation is moderate. Since upgrading each clone requires duplicating all over again, such recurring maintenance may be undesirable amount of work. Scripted inputs, relatively defined, will work natively.
  • Gotchas: Upgrades can be error prone and labor intensive, since you effectively recreate each clone. You may also need a naming convention to identify the source and its various clones. Hard-coded references to the original app name will cause errors, since the original app is not deployed.

Parent and children

  • Approach: Deploy the app with a configuration that is common to all clients and call it the 'parent'. Create 'children' that contain only the deviations needed for groups of clients. Deploy the children to their targets.
  • Level of effort: Moderate to large. Since all configuration is centralized, this approach scales well to clients, but becomes difficult to corral as the number of children increases (get it?).
  • Gotchas: Sharing scripted inputs in this model causes many other changes in order to work and therefore is not recommended. Also requires a naming convention to identify all the children and associate them with their parents. For example, when creating a variation of the Splunk_TA_windows from Splunkbase for my AWS environment, I'll name the variation Splunk_TA_windows-aws, which identifies it as a child of the parent app Splunk_TA_windows.

Symbolic links

  • Approach: Create symbolic links (symlinks) of all folders except local into a new app. In the new app, populate the local folder with the desired configuration for the target clients. The new app will now contain all of the substance of the original app, but with your local additions. Deploy only the new app, not the original app since it will be redundant and unused.
  • Level of effort: Implementation is complex. Upgrading is trivial.
  • Gotchas: There are passionate debates about symlinks because of the obfuscation introduced by the symlink abstraction. You may need a naming convention to identify the source and its various clones. Hard-coded references to the original app name will cause errors, since the original app is not deployed.

More?

Can you think of others? Anything else worth adding or considerations of these approaches?

adukes_splunk
Splunk Employee
Splunk Employee

Added related video.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Update: changed recommendation of Parent and children approach to discourage it's use when scripted inputs involved. In practice, the scripted inputs require hard coded fully qualified paths and hardcoded source and sourcetypes and indexes for this to work. In total, this amount of change renders the Parent and Child solution as a bad practice when scripted inputs are involved.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...