All Apps and Add-ons

What are the best practices to create a local configuration file?

sloshburch
Splunk Employee
Splunk Employee

While I can copy and paste and manually remove lines, this is laborious and has high chance for human error.

Copying entire stanzas, or the entire file, from the version in default may seem excessive. However, it duplicates the stanza's attributes and increases the potential to overlook changes made during upgrades of the TA.

A great example is with the Splunk Add-on for Unix and Linux. The documentation encourages you to copy the stanza and modify the disabled attribute. Also, you must add an index = attribute to each stanza to update the destination index. There are many inputs to manage and seems impractical. See Upgrade the Splunk Add-on for UNIX and Linux.

This example has a setup page through which Splunk create the local configuration file. See Enable data and scripted inputs for the Splunk Add-on for Unix and Linux.

Unfortunately, that solution applies only to inputs.conf and this particular add-on but is not a guaranteed solution for the more general case, especially in a highly distributed environment. See Scale your deployment with Splunk Enterprise components.

Are there any best practices for how to create such a local configuration file?

0 Karma
1 Solution

sloshburch
Splunk Employee
Splunk Employee

The Splunk Product Best Practices team provided this response. Read more about How Crowdsourcing is Shaping the Future of Splunk Best Practices.

Since you are using the Splunk Add-on for Unix and Linux as an example, it is safe to assume you have access to a UNIX shell. That's great because we can use it, with regular expressions, to generate a local version of the configuration with just the attributes you need. This solution extends beyond just inputs.conf and is available to use with any of the Splunk .conf files you need to make local.

If you are working with a Windows host, you could still access a UNIX shell by Running Linux on Windows or by using Cygwin. See Windows Subsystem for Linux Installation Guide for Windows 10 on the Microsoft website and Cygwin from the Cygwin website.

Solution

APP_DIR=""
CONF_NAME=""
ATTS_ADD=""
cd ${APP_DIR} && mkdir -p local && sed -rn 's/(^\[.+$)/\1\n\'"${ATTS_ADD}"'/p' default/${CONF_NAME}.conf >> local/${CONF_NAME}.conf && more local/${CONF_NAME}.conf

Explanation

Variables

The first few lines have some variables you can set:
- APP_DIR: Set this to where the application is installed within Splunk
- CONF_NAME: Set this to the configuration file name without file extension
- ATTS_ADD: List out any text you want added to each stanza using escape characters as desired. Inserting tabs may require ctrl+v and the tab key rather than \t.

Navigation

cd ${APP_DIR} && mkdir -p local &&

This portion navigates to the location on the file system and creates the local folder if it didn't already exist. The double ampersands ensure that consecutive steps run only if the current step is successful.

Regular Expression

sed -rn 's/(^\[.+$)/\1\n\'"${ATTS_ADD}"'/p' default/${CONF_NAME}.conf

The sed command does the work for us. The parameters -rn ensure the command outputs only that our fancy regular expressions are supported r, and that the original file is not printed n. On your UNIX command line, you can learn more within the sed manual: man sed.

The portion between the single quotes is of the general format 's/find/replace/p' where the leading s indicates substitution and the trailing p prints the replacement.

The "find" part of the regular expression is (^\[.+$). This creates a capture group ( ) of text that starts with a bracket, the Splunk stanza notation ^\[ followed by any characters .+ to the end of the line $.

The "replace" portion of the regular expression is \1\n\'"${ATTS_ADD}"'. This prints the capture group \1, a newline \n, and the output of what we set in the ATTS_ADD variable.

Output

>> local/${CONF_NAME}.conf

Send the output >> to a file. This will append, while a single greater-than > will overwrite any existing file.

View the Results

more local/${CONF_NAME}.conf

Now you have a clean new configuration file to edit and toggle any attributes or further customize to meet your needs!

Example

Since you understand the logic, you can test this solution in your environment.


APP_DIR="${SPLUNK_HOME}/etc/apps/Splunk_TA_nix"
CONF_NAME="inputs"
ATTS_ADD="disabled = true\nindex = os\n"
cd ${APP_DIR} && mkdir -p local && sed -rn 's/(^[.+$)/\1\n\'"${ATTS_ADD}"'/p' default/${CONF_NAME}.conf >> local/${CONF_NAME}.conf && more local/${CONF_NAME}.conf

View solution in original post

0 Karma

jeffland
SplunkTrust
SplunkTrust

As indicated by @jkat54's comments below your answer, the question "How should I create local config?" seems to touch more than one area where best practices can be applied. I'll try to chime in on the subjects I can think of off the top of my head.

Regarding "Which settings should go where?", I would say best practice is having all necessary settings in /default of an app with only selected changes kept in /local of that app. Obviously, this is because having a full copy of /default settings in /local will make your app insensitive to updates supplied by the app vendor. After all, it entirely defeats the purpose of local and default settings. Might sound obvious to anyone who's been using splunk for a while, but I would consider that a best practice anyone new to this topic should know about.

Regarding "How do I actually create the settings I need?", whether you feel comfortable writing your own from scratch or straight copy them from default and remove or modify settings as needed depends on your experience and does not really make a difference in my opinion. Of course you can also use helpers such as the script provided in @SloshBurch's answer if that helps you. What I would recommend in any case and would consider best practice is using btool's check method to avoid obvious errors: splunk btool check will check the config for your local instance, but you can easily use it for apps in deployment with a little trick I've taken from @sowings_splunk here:

mkdir -p /tmp/btool/apps
cp -r %SPLUNK_HOME/etc/deplyoment-apps/* /tmp/btool/apps
%SPLUNK_HOME/bin/splunk btool check --dir=/tmp/btool

This catches any typos in stanza and setting names, though it can't actually check whether your settings are correct. It might make sense to consider adding this to your build routine or have this run automatically at any other stage before deployment, but that's stuff for a different best practice post. I totally agree with jkat54's comments regarding code repositories and automation in the deployment process.

There are more options to consider regarding "Where do I keep my settings?". For example, it might make sense to have certain settings in a different app entirely. Going with the example of the Splunk Add-on for Unix and Linux, you might want to have certain settings enabled on certain hosts. For that, you could enable these each in their own app, that is, two separate apps, each with their own inputs.conf, that you deploy with different server classes alongside the Add-On. This seems to already have its own thread in this other best practices post.

sloshburch
Splunk Employee
Splunk Employee

My hero! Great contribution!

0 Karma

sloshburch
Splunk Employee
Splunk Employee

The Splunk Product Best Practices team provided this response. Read more about How Crowdsourcing is Shaping the Future of Splunk Best Practices.

Since you are using the Splunk Add-on for Unix and Linux as an example, it is safe to assume you have access to a UNIX shell. That's great because we can use it, with regular expressions, to generate a local version of the configuration with just the attributes you need. This solution extends beyond just inputs.conf and is available to use with any of the Splunk .conf files you need to make local.

If you are working with a Windows host, you could still access a UNIX shell by Running Linux on Windows or by using Cygwin. See Windows Subsystem for Linux Installation Guide for Windows 10 on the Microsoft website and Cygwin from the Cygwin website.

Solution

APP_DIR=""
CONF_NAME=""
ATTS_ADD=""
cd ${APP_DIR} && mkdir -p local && sed -rn 's/(^\[.+$)/\1\n\'"${ATTS_ADD}"'/p' default/${CONF_NAME}.conf >> local/${CONF_NAME}.conf && more local/${CONF_NAME}.conf

Explanation

Variables

The first few lines have some variables you can set:
- APP_DIR: Set this to where the application is installed within Splunk
- CONF_NAME: Set this to the configuration file name without file extension
- ATTS_ADD: List out any text you want added to each stanza using escape characters as desired. Inserting tabs may require ctrl+v and the tab key rather than \t.

Navigation

cd ${APP_DIR} && mkdir -p local &&

This portion navigates to the location on the file system and creates the local folder if it didn't already exist. The double ampersands ensure that consecutive steps run only if the current step is successful.

Regular Expression

sed -rn 's/(^\[.+$)/\1\n\'"${ATTS_ADD}"'/p' default/${CONF_NAME}.conf

The sed command does the work for us. The parameters -rn ensure the command outputs only that our fancy regular expressions are supported r, and that the original file is not printed n. On your UNIX command line, you can learn more within the sed manual: man sed.

The portion between the single quotes is of the general format 's/find/replace/p' where the leading s indicates substitution and the trailing p prints the replacement.

The "find" part of the regular expression is (^\[.+$). This creates a capture group ( ) of text that starts with a bracket, the Splunk stanza notation ^\[ followed by any characters .+ to the end of the line $.

The "replace" portion of the regular expression is \1\n\'"${ATTS_ADD}"'. This prints the capture group \1, a newline \n, and the output of what we set in the ATTS_ADD variable.

Output

>> local/${CONF_NAME}.conf

Send the output >> to a file. This will append, while a single greater-than > will overwrite any existing file.

View the Results

more local/${CONF_NAME}.conf

Now you have a clean new configuration file to edit and toggle any attributes or further customize to meet your needs!

Example

Since you understand the logic, you can test this solution in your environment.


APP_DIR="${SPLUNK_HOME}/etc/apps/Splunk_TA_nix"
CONF_NAME="inputs"
ATTS_ADD="disabled = true\nindex = os\n"
cd ${APP_DIR} && mkdir -p local && sed -rn 's/(^[.+$)/\1\n\'"${ATTS_ADD}"'/p' default/${CONF_NAME}.conf >> local/${CONF_NAME}.conf && more local/${CONF_NAME}.conf

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Updated to fix some slashes in the direction and include guidance on how to insert tabs.

0 Karma

jkat54
SplunkTrust
SplunkTrust

I severely hesitate to recommend sed and such complexities as a best practice.

In my opinion this topic unfolds into several scenarios and options.

1st being, you should have a code repository for all your splunk config files.
2nd being, anything you have in production should also be in said code repository
3rd, before deploying to prod, check new apps/code into code repo, and do the laborious reverse engineering and customization needed to integrate code into YOUR environment.

Now, how to get from code repo to production will be up to you.

You might use git, you might use TFS or visual studio online, or you could be like most devops shops and use a deployment plan. If you use a deployment plan and choose MS Word for your editor, i highly recommend putting all CODE SNIPPETS into single cell tables. I also recommend disabling auto-correct features such as the ones that turn double hyphen into one long dash, etc. Yes this is fraught with human error, so best approach is to automate the repo -> production pipeline as much as possible using tools available to you.

Final note, always backup first as part of your standard deployment process 🙂

0 Karma

sloshburch
Splunk Employee
Splunk Employee

@jkat54 strikes again with great feedback!

If you consider this topic oriented specifically around how you would create the config files for non-prod that would get checked into source control, what are your thoughts?
In other words, I agree that folks should think twice before messing with this in production. This content is targeted for creating the file before deployment.

Thanks again!

0 Karma

jkat54
SplunkTrust
SplunkTrust

You asked for it! Ima deliver it!

So yeah if I look at it that way, it’s more of a ,
“Thanks for sharing your approach” reaction from me 😉

As best practice it scares me.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...