Deployment Architecture
Highlighted

What is the best way to remove local config in a Search Head Cluster?

Ultra Champion

Scenario:
Anyone using a SHC (Search Head Cluster) implements apps from the Deployer. The deployer collapses the local and default config directories into default and pushes the config to the SHC members.

After normal usage, some of the knowledge objects in the app have evolved (like a savedsearch or a macro has been modified).

Eventually a new version of the app comes out and it has a butt kickin' nice new version of that very knowledge object. So, I stage the new version of the app on the deployer and push it out.

Unfortunately, the local folder edit of the knowledge object still takes precedent and the sweet new version (sitting in the default directory on the SHC members) is ignored.

How do we eliminate our version of the config and revert back to the one in the local directory?

Since we can't delete the edited version of the knowledge object from the UI, and we can't manually edit the conf file, what is the recommended way to address this?

More detail:
If you use the Deployer to send the splunkappaws to your Search Head Cluster you'll then have a bunch of cool knowledge objects that you can edit. Let's pretend I want to edit aws-accesslog-sourcetype(1) to adjust it for my environment. Before the edit, this config lives ONLY in $SPLUNK_HOME/etc/apps/splunk_app_aws/default/macros.conf on the Search Head Cluster Members. I make the my change in the UI and the result is this:
alt text
Notice my new definition of blah but no way to delete or revert back. There is now a corresponding version of this macro on the Search Head Cluster Members in $SPLUNK_HOME/etc/apps/splunk_app_aws/local/macros.conf defined as blah.

Now let's pretend that after some time, I want to remove my change and go back to the version provided in $SPLUNK_HOME/etc/apps/splunk_app_aws/default/macros.conf - with a single search head OR a search head pool, I can simply remove the corresponding stanza in $SPLUNK_HOME/etc/apps/splunk_app_aws/local/macros.conf with a text editor and restart the instance thereby allowing the version in $SPLUNK_HOME/etc/apps/splunk_app_aws/default/macros.conf to take affect.

Unfortunately, you cannot make manual edits to configuration in a Search Head Cluster. So is there a parallel way to remove your $SPLUNK_HOME/etc/apps/splunk_app_aws/local/macros.conf version in a Search Head Cluster?

Highlighted

Re: What is the best way to remove local config in a Search Head Cluster?

SplunkTrust
SplunkTrust

You should be able to delete the edited version from the UI, and it will not delete the version sitting in default.

Another method might be to "blank out" the local version via editing using the UI.

In the backend this would "mash" together like this:

Default savedsearches.conf:
[awesomeSearch]
awesomeNewSettings
...
more new greatSTuff

Local savedsearches.conf:
[awesomeSearch]
...nothing here...

When those mash together at "read config time" you'd only get the "awesomeNewSettings".

But does this all work? I cant test right now, so it's purely theory.

0 Karma
Highlighted

Re: What is the best way to remove local config in a Search Head Cluster?

Ultra Champion

Nah, you can't delete from the UI. For example, you can edit a macro that comes from the AWS app and the UI doesn't provide a way to delete your edit to revert back to the default version.

Setting a value to blank in the UI would leave an entry in the conf file but with no value - effectively unsetting the parameter. This would still override the version in default with a value of nothing.

Highlighted

Re: What is the best way to remove local config in a Search Head Cluster?

Splunk Employee
Splunk Employee

First, I assume you meant to say default here: How do we eliminate our version of the config and revert back to the one in the local directory?

If you want to remove all the local conf files here is a simple command line that will do it very quickly on a search head.

WARNING: backup of etc/apps recommended first!

splunk btool check --debug | egrep ^Checking.+/apps.+/local | sed -e 's/Checking: //' | xargs rm

You could add another grep in the list if you want to limit it to only certain apps.

Be careful, good luck!

0 Karma
Highlighted

Re: What is the best way to remove local config in a Search Head Cluster?

Ultra Champion

Nope. I mean to ask how we delete our change (in local) and revert back to the on that came out of the box (in default).

You can't edit or delete conf files in a Search Head Cluster - it will render the Cluster out of sync. I tried to articulate that in the first post with this "Since we can't delete the edited version of the knowledge object from the UI, and we can't manually edit the conf file, what is the recommended way to address this?" I'll try to edit the post to be more clear.

0 Karma
Highlighted

Re: What is the best way to remove local config in a Search Head Cluster?

Splunk Employee
Splunk Employee

Your right, you have to stop all of the SHC members, run those deletes then start the cluster.

0 Karma
Highlighted

Re: What is the best way to remove local config in a Search Head Cluster?

Ultra Champion

Wouldn't that still break the RAFT sync for the SHC?

0 Karma
Highlighted

Re: What is the best way to remove local config in a Search Head Cluster?

Ultra Champion

This also doesn't work with a REST command:

$ curl -k --request DELETE https://host:port/servicesNS/nobody/splunk_app_aws/configs/conf-macros/aws-accesslog-sourcetype%281%...

<msg type="ERROR">Object id=aws-accesslog-sourcetype(1) cannot be deleted in config=macros.</msg>
Highlighted

Re: What is the best way to remove local config in a Search Head Cluster?

Super Champion

I have asked atleast 5 questions on same issues. This default and local is such a hardwork on clusters and I hope Splunk will look into this

0 Karma
Highlighted

Re: What is the best way to remove local config in a Search Head Cluster?

Contributor

The ways I can think of would be to either:
1. Stop all SHC members, remove the local folder for the app, start them up again.
2. Move the app off the SHC deployer, let the SHC delete the app from themselves, redeploy.
3. I've never tried this, but maybe it could work: instead of using the SHC deployer, deploy everything to the SHC captain from a regular deployment server or manually and see if it'll replicate everything across the rest of the members anyways.

0 Karma