All Apps and Add-ons

How to have env specific behavior for apps?

aprummer
Explorer

We currently use an automatic deployment for splunk-apps. We create a tar-archive of the app and deploy it via $SPLUNK_HOME/bin/splunk install app app.tar -update 1 (currently only single-machine-setup).

  • For apps using scripts we have a .env file, which is created initially and loaded via the scripts.
  • not sure how I can make e.g. savedsearches.conf configurable. Env variable replacements like action.email.to = $ALTER_EMAIL don't seem to work.
  • using the local files for this wouldn't allow changes to the alert configuration.

Is there a best practice advice?

0 Karma
1 Solution

aprummer
Explorer

Also based on @SloshBurch and @feth inputs, I came to the following conclusion:

Prerequisites
- We want to have one artifact, which can be deployed to multiple environments
- We want to minimize custom logic to apply env variables
- We want to easily check if all required env variables are in place

Solution
- we build one single tar file for an app (it's the same for all environments)
- before deploy we check if all env variables are in place (currently we use a deployment repo with configurations and jenkins to store our secrets). This is checked via a groovy script
- we deploy our current version to the splunk instance, deleting all local-files (see remarks below)
- we assemble a rest call via groovy and execute it against our splunk instance (see remarks below)
- we restart the splunk instance

Remarks
For some apps/parts of apps it's good that you can configure them via the interface. For having control I want it in version control/a config-service. Deleting the local-files ensure that state. Unfortunately the ui doesn't seem to allow the sett configs only partially, like it's done with the api calls to /servicesNS/nobody/{{ splunkApp }}/properties/{{ splunkConfigPath }}.

For system wide config it's really nice to have config apps. But, ... I like to be more transparent about what app needs which config.

View solution in original post

aprummer
Explorer

Also based on @SloshBurch and @feth inputs, I came to the following conclusion:

Prerequisites
- We want to have one artifact, which can be deployed to multiple environments
- We want to minimize custom logic to apply env variables
- We want to easily check if all required env variables are in place

Solution
- we build one single tar file for an app (it's the same for all environments)
- before deploy we check if all env variables are in place (currently we use a deployment repo with configurations and jenkins to store our secrets). This is checked via a groovy script
- we deploy our current version to the splunk instance, deleting all local-files (see remarks below)
- we assemble a rest call via groovy and execute it against our splunk instance (see remarks below)
- we restart the splunk instance

Remarks
For some apps/parts of apps it's good that you can configure them via the interface. For having control I want it in version control/a config-service. Deleting the local-files ensure that state. Unfortunately the ui doesn't seem to allow the sett configs only partially, like it's done with the api calls to /servicesNS/nobody/{{ splunkApp }}/properties/{{ splunkConfigPath }}.

For system wide config it's really nice to have config apps. But, ... I like to be more transparent about what app needs which config.

sloshburch
Splunk Employee
Splunk Employee

Nice job putting it all together into a single clear answer!

0 Karma

sloshburch
Splunk Employee
Splunk Employee

If you use source control (git) you could replace the first step with a push/pull.

The way I used to do this was to have my lab contain all the configuration. And part of promoting from nonprod to prod was checking in and push from nonprod to sourcecontrol. Then in prod, I would pull (replacing everything - risky) and use some perl and sed commands to find lines of config immediately below #NONPROD and insert a comment hash at the start of their line (thereby disabling those nonprod specific env items. Then another command that did the inverse and removed comments preceding the lines that fall just below #PROD.

I personally enjoyed this because all config was in the same place. No values kept elsewhere for RESP api calls etc...

0 Karma

aprummer
Explorer

thanks for your elaboration.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

I may be low on coffee, but I keep reading this and I'm having trouble feeling confident that I'm understanding.

It sounds like there's an automatic process for deploying apps but you want the apps to have slight adjustments based on environmental differences which you capture in a .env. file.

It's hard to understand how the env file is processed and how it applies to the in-app files.

Stepping back a bit, is the goal to have golden config repo so you can promote up from non prod to prod without differences in app? If so, my gut is to start by isolating all env specific details into a single app that IS different between environments (duh) and that allows you to minimize any mods in the apps that remain.

I'll elaborate if that approach is promising.

0 Karma

aprummer
Explorer

Low on coffee, but to the point.

Talking to our IT-consultants and a teammate who did the splunk operation course, I was pointed to this practice as well. Together with the possibility to change partial config settings* via the api, I'm coming close to a smooth solution. In my mind I would do the following way:

  • package config app
  • deploy to env
  • run api calls (this allows config/secrets providers like conjur/consul)
  • updated restart splunk

I would be happy if you can elaborate your approach. Thank you

... *) I didn't know that this is possible. My first try with partial local files failed

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...