Splunk Search

merging local into default using git

Motivator

I would like to periodically merge stuff in /local into /default and then delete whatever is in /local
I have a repository per app and a remote git server
I think it should be possible to merge these two subfolders with /local winning over /default but i am severely lacking in git knowledge so please be gentle 🙂

clarification:
The idea is to merge UI generated changes back into default (on INTG), push it to the central git and then pull the repo version to a splunk PROD stage ( /etc/apps or /etc/deployment-apps or /etc/cluster-apps or whatever was the path again).

Mostly for app maintainers updating searches and knowledge objects on the GUI with no shell access.

Tags (2)
1 Solution

Motivator

If you face the same issue just use the new custom command mergeapp in https://splunkbase.splunk.com/app/3685/

View solution in original post

0 Karma

Motivator

I'm also looking into this as I have the use case of migrating apps that were created by internal users using the Splunk UI to Splunk Cloud. If we just package up the app as is and try to upload to Splunk Cloud, it gets rejected due to some config files being under local. Apps with a lot of objects can be very time consuming to merge manually.

One of my colleges Mittal came up with the creative solution of just taring up the app, copying it over to our test deployer, and deploying the app to the test search cluster. During that process, the deployer does the merge of local and default to default for you. Then your merged version is under SPLUNK_HOME/etc/apps on the search cluster member.

0 Karma

Super Champion

So I know I'm a bit late to the party here, but wanted to throw out another option that I've found useful. Now, as a disclaimer, I'm a bit biased since I'm the author.

I've written a tool called ksconf that we use in our consulting practice to handle this type of scenario for our clients. It handles several basic .conf file operations like merging, comparing, sorting, and syntax checking. But besides these simple operations, it has the ability to manage an arbitrary number of configuration layers (more than the 2 provided by Splunk - default & local); so issues like maintaining a long-term changes on top of an app that originated in Splunkbase and tracking both your customizations and upstream changes simultaneously is possible without turning the "default" folder into a version control nightmare. There's also a promotion feature that I use to take and move changes that I've made via the UI (which go into the local folder) and then "promote" those specific changes into a folder that's version controlled. (For me, this has cut down on mistakes introduced while editing both local and default files manually in vim.) The tool lets me pick and choose which bits are promoted and which ones are kept in local (because they are either specific to the local dev system, or because they are still being developed.)

Another great feature is pre-commit hook integration, so for example you can have all of your .conf files syntax checked before they get committed. This has saved me from myself on many occasions.

So you can use this tool to do simple things (much like what I suspect crudini can do), or you can integrate it into your git workflow, or use it to do much more extensive work like maintaining multiple layers of config files (giving you a "default.d" like feature), it's entirely up to you.

BTW, I have a talk at the Splunk User Community Experience during .conf2018 in a few days. Feel free to check it out.

Motivator

Hi, would love to talk about it some if you have time. I am at .conf already (university) - any chance to meet for chat?

0 Karma

Path Finder

As this is a top result on Google when searching for this issue, I'd like to offer an alternative way I found, using a 3rd party tool called crudini.

Splunk config files work well enough with that tool, so if you don't want to create a complete "packaged" app, but just combine two Splunk config files, with no restriction if they're in the same app, use crudini - at least for RedHat, it's available from the EPEL repo, and also available in Debian.

crudini works like this:

crudini --merge default/props.conf < local/props.conf

The result is a .conf file, that contains the data from the default props, updated and completed by whatever is available in the local props file. It can obviously also be used to set/change/delete values in Splunk conf files from within your scripts.

Disclaimer: I did not yet properly test it with all edge cases, e.g. multiline values. Use at your own discretion 😉

0 Karma

Motivator

Sounds cool, i can have a look at it. a pure python way would be nice...
I have a semi working git (pure python) for splunk but it is not yet totaly nice. if anyone is interested...

0 Karma

Path Finder

Well, for Python, check the ConfigParser package. 😉

0 Karma

Motivator

If you face the same issue just use the new custom command mergeapp in https://splunkbase.splunk.com/app/3685/

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

Isn't that a little light on cleaning up /local? E.g. eventtypes.conf, tags.conf, fields.conf, etc.

0 Karma

Motivator

i updated it for .conf - it just checks what .spec files are under system and clears those.

0 Karma

Motivator

yeah i think this would be a better default set:
(from line 47:)

os.remove(os.path.join(appsPath,appName,'local','alertactions.conf'))
os.remove(os.path.join(appsPath,appName,'local','authentication.conf'))
os.remove(os.path.join(appsPath,appName,'local','authorize.conf'))
os.remove(os.path.join(appsPath,appName,'local','commands.conf'))
os.remove(os.path.join(appsPath,appName,'local','database.conf'))
os.remove(os.path.join(appsPath,appName,'local','datamodels.conf'))
os.remove(os.path.join(appsPath,appName,'local','db
connectiontypes.conf'))
os.remove(os.path.join(appsPath,appName,'local','db
connections.conf'))
os.remove(os.path.join(appsPath,appName,'local','dblookup.conf'))
os.remove(os.path.join(appsPath,appName,'local','distsearch.conf'))
os.remove(os.path.join(appsPath,appName,'local','eventrenderers.conf'))
os.remove(os.path.join(appsPath,appName,'local','eventgen.conf'))
os.remove(os.path.join(appsPath,appName,'local','eventtypes.conf'))
os.remove(os.path.join(appsPath,appName,'local','fields.conf'))
os.remove(os.path.join(appsPath,appName,'local','identities.conf'))
os.remove(os.path.join(appsPath,appName,'local','indexes.conf'))
os.remove(os.path.join(appsPath,appName,'local','inputs.conf'))
os.remove(os.path.join(appsPath,appName,'local','java.conf'))
os.remove(os.path.join(appsPath,appName,'local','limits.conf'))
os.remove(os.path.join(appsPath,appName,'local','macros.conf'))
os.remove(os.path.join(appsPath,appName,'local','outputs.conf'))
os.remove(os.path.join(appsPath,appName,'local','props.conf'))
os.remove(os.path.join(appsPath,appName,'local','restmap.conf'))
os.remove(os.path.join(appsPath,appName,'local','savedsearches.conf'))
os.remove(os.path.join(appsPath,appName,'local','searchbnf.conf'))
os.remove(os.path.join(appsPath,appName,'local','server.conf'))
os.remove(os.path.join(appsPath,appName,'local','settings.conf'))
os.remove(os.path.join(appsPath,appName,'local','tags.conf'))
os.remove(os.path.join(appsPath,appName,'local','telemetry.conf'))
os.remove(os.path.join(appsPath,appName,'local','transforms.conf'))
os.remove(os.path.join(appsPath,appName,'local','ui-prefs.conf'))
os.remove(os.path.join(appsPath,appName,'local','user-prefs.conf'))
os.remove(os.path.join(appsPath,appName,'local','viewstates.conf'))
os.remove(os.path.join(appsPath,appName,'local','visualizations.conf'))
os.remove(os.path.join(appsPath,appName,'local','web.conf'))
os.remove(os.path.join(appsPath,appName,'local','workflow
actions.conf'))

0 Karma

SplunkTrust
SplunkTrust

You can use Splunk for the merge, call the /services/apps/local/<name>/package endpoint to generage a .spl file with local and default merged together into default.

To be lazy, you can use the App Exporter: https://splunkbase.splunk.com/app/2613/

0 Karma

SplunkTrust
SplunkTrust

It makes sense, I'm just not aware of something apart from the package endpoint that has the local-to-default merge logic already built.

0 Karma

Motivator

ok, so i ended up doing my custom command and its up for grabs at https://splunkbase.splunk.com/app/3685/

0 Karma

SplunkTrust
SplunkTrust

I don't see why not. Package app, put content of .spl wherever you commit to git from, commit to git.

0 Karma

Motivator

hm, yeah kinda look ok-ish. The idea was to work with a repository equaling an app and work directly with it in /etc/apps for instance.

The general workflow is supposed to be
1. fork app from template in bitbucket
2. clone repo into /etc/apps (or /etc/deployment-apps or /etc/master-apps) on a development stage
3. edit the app, in the case of an ui app via splunk web-ui (power users)
4. merge changes, clean stuff in local
5. push changes into git
6. clone or update app into a production stage into the apropriate structure and refresh leading to usable app in ui or app being deployed via cluster or deployment method

we keep a separate gitdir so there is no git cluter in the repo itself.

Do i make sense?

I guess i can do what you suggest and take the "detour" via /package of which i was not aware

0 Karma

Motivator

ok, but this won't touch the app itself so i can not work with git directly etc.

0 Karma

Motivator

It is generally not recommended to have your local changes in the default folder, as any App Update will break your individual settings.

Are there any special reasons for you to not keep the local configs?

0 Karma

Motivator

The idea is to merge UI generated changes back into default (on INTG), push it to the central git and then pull the repo version to a splunk PROD stage ( /etc/apps or /etc/deployment-apps or /etc/cluster-apps or whatever was the path again).

Mostly for app maintainers updating searches and knowledge objects on the GUI with no shell access.

I will update the question to clarify

0 Karma

Super Champion

hi mate, why you merge into "default" ? Better to keep within "local" of your app.
You doing this for SH cluster?

Also the merging shouldn't be done using git, but you should look into btool

0 Karma

Motivator

open for pointers

essentially i want to merge all changes by non admins with shell and vi 🙂 into the default

0 Karma