Merging local into default using git?

[dominiquevocat]

I would like to periodically merge stuff in /local into /default and then delete whatever is in /local
I have a repository per app and a remote git server
I think it should be possible to merge these two subfolders with /local winning over /default but i am severely lacking in git knowledge so please be gentle 🙂

clarification:
The idea is to merge UI generated changes back into default (on INTG), push it to the central git and then pull the repo version to a splunk PROD stage ( /etc/apps or /etc/deployment-apps or /etc/cluster-apps or whatever was the path again).

Mostly for app maintainers updating searches and knowledge objects on the GUI with no shell access.

[jotne]

App Merger app seems to not work in Splunk 9.x  (may be python 2?)

Can you test and fix?

[bandit]

I'm also looking into this as I have the use case of migrating apps that were created by internal users using the Splunk UI to Splunk Cloud. If we just package up the app as is and try to upload to Splunk Cloud, it gets rejected due to some config files being under local. Apps with a lot of objects can be very time consuming to merge manually.

One of my colleges Mittal came up with the creative solution of just taring up the app, copying it over to our test deployer, and deploying the app to the test search cluster. During that process, the deployer does the merge of local and default to default for you. Then your merged version is under SPLUNK_HOME/etc/apps on the search cluster member.

[Lowell]

So I know I'm a bit late to the party here, but wanted to throw out another option that I've found useful. Now, as a disclaimer, I'm a bit biased since I'm the author.

I've written a tool called ksconf that we use in our consulting practice to handle this type of scenario for our clients. It handles several basic .conf file operations like merging, comparing, sorting, and syntax checking. But besides these simple operations, it has the ability to manage an arbitrary number of configuration layers (more than the 2 provided by Splunk - default & local); so issues like maintaining a long-term changes on top of an app that originated in Splunkbase and tracking both your customizations and upstream changes simultaneously is possible without turning the "default" folder into a version control nightmare. There's also a promotion feature that I use to take and move changes that I've made via the UI (which go into the local folder) and then "promote" those specific changes into a folder that's version controlled. (For me, this has cut down on mistakes introduced while editing both local and default files manually in vim.) The tool lets me pick and choose which bits are promoted and which ones are kept in local (because they are either specific to the local dev system, or because they are still being developed.)

Another great feature is pre-commit hook integration, so for example you can have all of your .conf files syntax checked before they get committed. This has saved me from myself on many occasions.

So you can use this tool to do simple things (much like what I suspect crudini can do), or you can integrate it into your git workflow, or use it to do much more extensive work like maintaining multiple layers of config files (giving you a "default.d" like feature), it's entirely up to you.

BTW, I have a talk at the Splunk User Community Experience during .conf2018 in a few days. Feel free to check it out.

[dominiquevocat]

Hi, would love to talk about it some if you have time. I am at .conf already (university) - any chance to meet for chat?

[bjoernhansen]

As this is a top result on Google when searching for this issue, I'd like to offer an alternative way I found, using a 3rd party tool called crudini.

Splunk config files work well enough with that tool, so if you don't want to create a complete "packaged" app, but just combine two Splunk config files, with no restriction if they're in the same app, use crudini - at least for RedHat, it's available from the EPEL repo, and also available in Debian.

crudini works like this:

crudini --merge default/props.conf < local/props.conf

The result is a .conf file, that contains the data from the default props, updated and completed by whatever is available in the local props file. It can obviously also be used to set/change/delete values in Splunk conf files from within your scripts.

Disclaimer: I did not yet properly test it with all edge cases, e.g. multiline values. Use at your own discretion 😉

[dominiquevocat]

Sounds cool, i can have a look at it. a pure python way would be nice...
I have a semi working git (pure python) for splunk but it is not yet totaly nice. if anyone is interested...

[bjoernhansen]

Well, for Python, check the ConfigParser package. 😉

[dominiquevocat · Accepted solution]

If you face the same issue just use the new custom command mergeapp in https://splunkbase.splunk.com/app/3685/

[martin_mueller]

Isn't that a little light on cleaning up /local? E.g. eventtypes.conf, tags.conf, fields.conf, etc.

[dominiquevocat]

i updated it for .conf - it just checks what .spec files are under system and clears those.

[dominiquevocat]

yeah i think this would be a better default set:
(from line 47:)

os.remove(os.path.join(appsPath,appName,'local','alert_actions.conf'))
os.remove(os.path.join(appsPath,appName,'local','authentication.conf'))
os.remove(os.path.join(appsPath,appName,'local','authorize.conf'))
os.remove(os.path.join(appsPath,appName,'local','commands.conf'))
os.remove(os.path.join(appsPath,appName,'local','database.conf'))
os.remove(os.path.join(appsPath,appName,'local','datamodels.conf'))
os.remove(os.path.join(appsPath,appName,'local','db_connection_types.conf'))
os.remove(os.path.join(appsPath,appName,'local','db_connections.conf'))
os.remove(os.path.join(appsPath,appName,'local','dblookup.conf'))
os.remove(os.path.join(appsPath,appName,'local','distsearch.conf'))
os.remove(os.path.join(appsPath,appName,'local','event_renderers.conf'))
os.remove(os.path.join(appsPath,appName,'local','eventgen.conf'))
os.remove(os.path.join(appsPath,appName,'local','eventtypes.conf'))
os.remove(os.path.join(appsPath,appName,'local','fields.conf'))
os.remove(os.path.join(appsPath,appName,'local','identities.conf'))
os.remove(os.path.join(appsPath,appName,'local','indexes.conf'))
os.remove(os.path.join(appsPath,appName,'local','inputs.conf'))
os.remove(os.path.join(appsPath,appName,'local','java.conf'))
os.remove(os.path.join(appsPath,appName,'local','limits.conf'))
os.remove(os.path.join(appsPath,appName,'local','macros.conf'))
os.remove(os.path.join(appsPath,appName,'local','outputs.conf'))
os.remove(os.path.join(appsPath,appName,'local','props.conf'))
os.remove(os.path.join(appsPath,appName,'local','restmap.conf'))
os.remove(os.path.join(appsPath,appName,'local','savedsearches.conf'))
os.remove(os.path.join(appsPath,appName,'local','searchbnf.conf'))
os.remove(os.path.join(appsPath,appName,'local','server.conf'))
os.remove(os.path.join(appsPath,appName,'local','settings.conf'))
os.remove(os.path.join(appsPath,appName,'local','tags.conf'))
os.remove(os.path.join(appsPath,appName,'local','telemetry.conf'))
os.remove(os.path.join(appsPath,appName,'local','transforms.conf'))
os.remove(os.path.join(appsPath,appName,'local','ui-prefs.conf'))
os.remove(os.path.join(appsPath,appName,'local','user-prefs.conf'))
os.remove(os.path.join(appsPath,appName,'local','viewstates.conf'))
os.remove(os.path.join(appsPath,appName,'local','visualizations.conf'))
os.remove(os.path.join(appsPath,appName,'local','web.conf'))
os.remove(os.path.join(appsPath,appName,'local','workflow_actions.conf'))

[martin_mueller]

You can use Splunk for the merge, call the /services/apps/local/<name>/package endpoint to generage a .spl file with local and default merged together into default.

To be lazy, you can use the App Exporter: https://splunkbase.splunk.com/app/2613/

[martin_mueller]

It makes sense, I'm just not aware of something apart from the package endpoint that has the local-to-default merge logic already built.

[dominiquevocat]

ok, so i ended up doing my custom command and its up for grabs at https://splunkbase.splunk.com/app/3685/

[martin_mueller]

I don't see why not. Package app, put content of .spl wherever you commit to git from, commit to git.

[dominiquevocat]

hm, yeah kinda look ok-ish. The idea was to work with a repository equaling an app and work directly with it in /etc/apps for instance.

The general workflow is supposed to be
1. fork app from template in bitbucket
2. clone repo into /etc/apps (or /etc/deployment-apps or /etc/master-apps) on a development stage
3. edit the app, in the case of an ui app via splunk web-ui (power users)
4. merge changes, clean stuff in local
5. push changes into git
6. clone or update app into a production stage into the apropriate structure and refresh leading to usable app in ui or app being deployed via cluster or deployment method

we keep a separate gitdir so there is no git cluter in the repo itself.

Do i make sense?

I guess i can do what you suggest and take the "detour" via /package of which i was not aware

[dominiquevocat]

ok, but this won't touch the app itself so i can not work with git directly etc.

[DMohn]

It is generally not recommended to have your local changes in the default folder, as any App Update will break your individual settings.

Are there any special reasons for you to not keep the local configs?

[dominiquevocat]

The idea is to merge UI generated changes back into default (on INTG), push it to the central git and then pull the repo version to a splunk PROD stage ( /etc/apps or /etc/deployment-apps or /etc/cluster-apps or whatever was the path again).

Mostly for app maintainers updating searches and knowledge objects on the GUI with no shell access.

I will update the question to clarify

[koshyk]

hi mate, why you merge into "default" ? Better to keep within "local" of your app.
You doing this for SH cluster?

Also the merging shouldn't be done using git, but you should look into btool