Merging local into default using git?

[dominiquevocat]

I would like to periodically merge stuff in /local into /default and then delete whatever is in /local
I have a repository per app and a remote git server
I think it should be possible to merge these two subfolders with /local winning over /default but i am severely lacking in git knowledge so please be gentle 🙂

clarification:
The idea is to merge UI generated changes back into default (on INTG), push it to the central git and then pull the repo version to a splunk PROD stage ( /etc/apps or /etc/deployment-apps or /etc/cluster-apps or whatever was the path again).

Mostly for app maintainers updating searches and knowledge objects on the GUI with no shell access.

[jotne]

App Merger app seems to not work in Splunk 9.x  (may be python 2?)

Can you test and fix?

[bandit]

I'm also looking into this as I have the use case of migrating apps that were created by internal users using the Splunk UI to Splunk Cloud. If we just package up the app as is and try to upload to Splunk Cloud, it gets rejected due to some config files being under local. Apps with a lot of objects can be very time consuming to merge manually.

One of my colleges Mittal came up with the creative solution of just taring up the app, copying it over to our test deployer, and deploying the app to the test search cluster. During that process, the deployer does the merge of local and default to default for you. Then your merged version is under SPLUNK_HOME/etc/apps on the search cluster member.

[Lowell]

So I know I'm a bit late to the party here, but wanted to throw out another option that I've found useful. Now, as a disclaimer, I'm a bit biased since I'm the author.

I've written a tool called ksconf that we use in our consulting practice to handle this type of scenario for our clients. It handles several basic .conf file operations like merging, comparing, sorting, and syntax checking. But besides these simple operations, it has the ability to manage an arbitrary number of configuration layers (more than the 2 provided by Splunk - default & local); so issues like maintaining a long-term changes on top of an app that originated in Splunkbase and tracking both your customizations and upstream changes simultaneously is possible without turning the "default" folder into a version control nightmare. There's also a promotion feature that I use to take and move changes that I've made via the UI (which go into the local folder) and then "promote" those specific changes into a folder that's version controlled. (For me, this has cut down on mistakes introduced while editing both local and default files manually in vim.) The tool lets me pick and choose which bits are promoted and which ones are kept in local (because they are either specific to the local dev system, or because they are still being developed.)

Another great feature is pre-commit hook integration, so for example you can have all of your .conf files syntax checked before they get committed. This has saved me from myself on many occasions.

So you can use this tool to do simple things (much like what I suspect crudini can do), or you can integrate it into your git workflow, or use it to do much more extensive work like maintaining multiple layers of config files (giving you a "default.d" like feature), it's entirely up to you.

BTW, I have a talk at the Splunk User Community Experience during .conf2018 in a few days. Feel free to check it out.

[dominiquevocat]

Hi, would love to talk about it some if you have time. I am at .conf already (university) - any chance to meet for chat?

[bjoernhansen]

As this is a top result on Google when searching for this issue, I'd like to offer an alternative way I found, using a 3rd party tool called crudini.

Splunk config files work well enough with that tool, so if you don't want to create a complete "packaged" app, but just combine two Splunk config files, with no restriction if they're in the same app, use crudini - at least for RedHat, it's available from the EPEL repo, and also available in Debian.

crudini works like this:

crudini --merge default/props.conf < local/props.conf

The result is a .conf file, that contains the data from the default props, updated and completed by whatever is available in the local props file. It can obviously also be used to set/change/delete values in Splunk conf files from within your scripts.

Disclaimer: I did not yet properly test it with all edge cases, e.g. multiline values. Use at your own discretion 😉

[dominiquevocat]

Sounds cool, i can have a look at it. a pure python way would be nice...
I have a semi working git (pure python) for splunk but it is not yet totaly nice. if anyone is interested...

[bjoernhansen]

Well, for Python, check the ConfigParser package. 😉

[dominiquevocat · Accepted solution]

If you face the same issue just use the new custom command mergeapp in https://splunkbase.splunk.com/app/3685/

[martin_mueller]

Isn't that a little light on cleaning up /local? E.g. eventtypes.conf, tags.conf, fields.conf, etc.

[dominiquevocat]

i updated it for .conf - it just checks what .spec files are under system and clears those.

[dominiquevocat]

yeah i think this would be a better default set:
(from line 47:)

os.remove(os.path.join(appsPath,appName,'local','alert_actions.conf'))
os.remove(os.path.join(appsPath,appName,'local','authentication.conf'))
os.remove(os.path.join(appsPath,appName,'local','authorize.conf'))
os.remove(os.path.join(appsPath,appName,'local','commands.conf'))
os.remove(os.path.join(appsPath,appName,'local','database.conf'))
os.remove(os.path.join(appsPath,appName,'local','datamodels.conf'))
os.remove(os.path.join(appsPath,appName,'local','db_connection_types.conf'))
os.remove(os.path.join(appsPath,appName,'local','db_connections.conf'))
os.remove(os.path.join(appsPath,appName,'local','dblookup.conf'))
os.remove(os.path.join(appsPath,appName,'local','distsearch.conf'))
os.remove(os.path.join(appsPath,appName,'local','event_renderers.conf'))
os.remove(os.path.join(appsPath,appName,'local','eventgen.conf'))
os.remove(os.path.join(appsPath,appName,'local','eventtypes.conf'))
os.remove(os.path.join(appsPath,appName,'local','fields.conf'))
os.remove(os.path.join(appsPath,appName,'local','identities.conf'))
os.remove(os.path.join(appsPath,appName,'local','indexes.conf'))
os.remove(os.path.join(appsPath,appName,'local','inputs.conf'))
os.remove(os.path.join(appsPath,appName,'local','java.conf'))
os.remove(os.path.join(appsPath,appName,'local','limits.conf'))
os.remove(os.path.join(appsPath,appName,'local','macros.conf'))
os.remove(os.path.join(appsPath,appName,'local','outputs.conf'))
os.remove(os.path.join(appsPath,appName,'local','props.conf'))
os.remove(os.path.join(appsPath,appName,'local','restmap.conf'))
os.remove(os.path.join(appsPath,appName,'local','savedsearches.conf'))
os.remove(os.path.join(appsPath,appName,'local','searchbnf.conf'))
os.remove(os.path.join(appsPath,appName,'local','server.conf'))
os.remove(os.path.join(appsPath,appName,'local','settings.conf'))
os.remove(os.path.join(appsPath,appName,'local','tags.conf'))
os.remove(os.path.join(appsPath,appName,'local','telemetry.conf'))
os.remove(os.path.join(appsPath,appName,'local','transforms.conf'))
os.remove(os.path.join(appsPath,appName,'local','ui-prefs.conf'))
os.remove(os.path.join(appsPath,appName,'local','user-prefs.conf'))
os.remove(os.path.join(appsPath,appName,'local','viewstates.conf'))
os.remove(os.path.join(appsPath,appName,'local','visualizations.conf'))
os.remove(os.path.join(appsPath,appName,'local','web.conf'))
os.remove(os.path.join(appsPath,appName,'local','workflow_actions.conf'))

[martin_mueller]

You can use Splunk for the merge, call the /services/apps/local/<name>/package endpoint to generage a .spl file with local and default merged together into default.

To be lazy, you can use the App Exporter: https://splunkbase.splunk.com/app/2613/

[BTrust]

Hi @martin_mueller,

The App Exporter don’t seem to work anymore, can that be fixed?

PS. Seems like the app I just need 🙂

Many thanks in advance.

Cheers,
Bjarne

[martin_mueller]

I haven't looked at it in years. If you just need to package an app once, call the REST API: 

| rest splunk_server=local services/apps/local/{name}/package

https://docs.splunk.com/Documentation/Splunk/9.4.1/RESTREF/RESTapps#apps.2Flocal.2F.7Bname.7D.2Fpack...

[BTrust]

Hi @martin_mueller,

As I’ve tried to explain- the package part work perfectly, the interesting part begins after this stage: How to get the package out, when not having physical access to the box.

The benefits begins after the export😊

Are there any REST available that can accomplish this?

[martin_mueller]

The App Exporter never solved that part.

Back when it was written and working, the /package REST call would give you a URL to download the package. That URL (& the download) was removed from the REST call in Splunk Enterprise long ago.

[BTrust]

Any you are not aware of any workarounds to this "limitation" I suppose (except for directs access to the box)?

[martin_mueller]

My workaround would be "talk to the people who do have access".

[isoutamo]

You could try this https://github.com/paychex/Splunk.Conf19 if it helps you to get all KOs on files and then create app on your local environment?

[BTrust]

Thanks @isoutamo ,

This was a great and useful input, most appreciated👍