What is the best way to remove local config in a search head cluster?

[sloshburch]

Scenario:
Anyone using a SHC (Search Head Cluster) implements apps from the Deployer. The deployer collapses the local and default config directories into default and pushes the config to the SHC members.

After normal usage, some of the knowledge objects in the app have evolved (like a savedsearch or a macro has been modified).

Eventually a new version of the app comes out and it has a butt kickin' nice new version of that very knowledge object. So, I stage the new version of the app on the deployer and push it out.

Unfortunately, the local folder edit of the knowledge object still takes precedent and the sweet new version (sitting in the default directory on the SHC members) is ignored.

How do we eliminate our version of the config and revert back to the one in the local directory?

Since we can't delete the edited version of the knowledge object from the UI, and we can't manually edit the conf file, what is the recommended way to address this?

More detail:
If you use the Deployer to send the splunk_app_aws to your Search Head Cluster you'll then have a bunch of cool knowledge objects that you can edit. Let's pretend I want to edit aws-accesslog-sourcetype(1) to adjust it for my environment. Before the edit, this config lives ONLY in $SPLUNK_HOME/etc/apps/splunk_app_aws/default/macros.conf on the Search Head Cluster Members. I make the my change in the UI and the result is this:

Notice my new definition of blah but no way to delete or revert back. There is now a corresponding version of this macro on the Search Head Cluster Members in $SPLUNK_HOME/etc/apps/splunk_app_aws/local/macros.conf defined as blah.

Now let's pretend that after some time, I want to remove my change and go back to the version provided in $SPLUNK_HOME/etc/apps/splunk_app_aws/default/macros.conf - with a single search head OR a search head pool, I can simply remove the corresponding stanza in $SPLUNK_HOME/etc/apps/splunk_app_aws/local/macros.conf with a text editor and restart the instance thereby allowing the version in $SPLUNK_HOME/etc/apps/splunk_app_aws/default/macros.conf to take affect.

Unfortunately, you cannot make manual edits to configuration in a Search Head Cluster. So is there a parallel way to remove your $SPLUNK_HOME/etc/apps/splunk_app_aws/local/macros.conf version in a Search Head Cluster?

[sloshburch]

Yea, but I don't think any of us are really satisfied with those, right?

1. I have a feeling this is likely to produce a sync issue with RAFT and would not really be supported
2. Def would work, but before you move it off the deployer, you'd have to throw a copy of the SHC member's local folder on the version to get re-deployed from the deployer, otherwise you'd lose all the Knowledge Objects folks created
3. This sounds like bad-news-bears and I'm sure any problems would not be supported.

Essentially, option 2 is the best but I've also submitted a feature request for this. Essentially, the non-SHC usage of splunk allowed this scenario to be addressed trivially and as such, no deliberate feature needed to be created to support it.

[ndgold]

@SloshBurch, which option did you choose? I'm having the same issue, and am not sure how to proceed. I am tempted to stop all SCH members and delete the KOs. However, I worry about the sync issues you mentioned above. Thanks.

[DATEVeG]

We did some modifications in all of our SHC members directly in metadata/local.meta. Afterwards we issued a rolling restart. There were no sync issues.

[ndgold]

Thanks. I stopped splunk on each search head, removed my KOs, and restarted. It seems to have worked.

[sloshburch]

Phew. I bet it's gotten more stable over the years. Many can't afford to shut down all SH to do this but glad to hear it worked.

FWIW, there's new features for pushing config that could also be manipulated for clearing some local config by pushing deactivated or empty stanzas...I think. To be honest I haven't tried yet though.

Learn more in the docs at Choose a deployer push mode

[ndgold]

Thanks for the link.

[gjanders]

Thankyou for submitting the feature request, this is one of the most frustrating things relating to search head clustering. I've got quite a few enhancement requests open and I'm noticing the ones I'm finding on the forums are generally not implemented.

I've seen other companies ignore feature requests for years , I'm hoping Splunk is different!

[alisterwhipp]

If you edit a local config to be exactly the same as the deployed default, does that make the local go away?

[sloshburch]

No. I talked with some Splunkers internally and learned that some Knowledge Objects have that behavior but not all. For many types of Knowledge Objects it merely persists a local config of the string you entered regardless of if it matches what was originally in default. 😞

[alisterwhipp]

I did wonder. It would require exactly matching.
For dashboards what we've been doing is removing the dashboard on the deployment, deleting on the shc, then re-adding via deployer. This of course means that dashboard is actually missing in action for as long as it takes to re-deploy after deleting the local. And, I have no idea if this would work for anything stored in savedsearches.conf.

[gn694]

I just tried this but it did not work for me:
1. Deleted dashbaord (xml file) on deployer
2. Applied the shcluster-bundle
3. Went to delete the dashboard on a Search Head but the delete option is still not available.
4. Verified on each SHCluster member in the app's default directory that the dashboard (xml file) is no longer present. It is now only in the local directory.
5. I restarted Splunk on one of the Search heads and still can not delete the (now local only) dashboard there.

[gn694]

Update: I forgot to also delete the details for the dashboard in the metadata file on the Deployer. Once that was also removed I was able to delete the local copy.

[sloshburch]

How do you delete on the SHC? If you do it on the filesystem you run the risk of a config mismatch...

[goodsellt]

Yea they're all bad, but it's a side effect of the odd design choice to have a "deployment" master server which then can start having radically different versions from the captain and the other SHC members. Good idea for a feature request, seems like something they need to address in one way or the other.

[sloshburch]

This also doesn't work with a REST command:

$ curl -k --request DELETE https://host:port/servicesNS/nobody/splunk_app_aws/configs/conf-macros/aws-accesslog-sourcetype%281%...

<msg type="ERROR">Object id=aws-accesslog-sourcetype(1) cannot be deleted in config=macros.</msg>

[koshyk]

I have asked atleast 5 questions on same issues. This default and local is such a hardwork on clusters and I hope Splunk will look into this

[sjohnson_splunk]

First, I assume you meant to say default here: How do we eliminate our version of the config and revert back to the one in the local directory?

If you want to remove all the local conf files here is a simple command line that will do it very quickly on a search head.

WARNING: backup of etc/apps recommended first!

splunk btool check --debug | egrep ^Checking.+/apps.+/local | sed -e 's/Checking: //' | xargs rm

You could add another grep in the list if you want to limit it to only certain apps.

Be careful, good luck!

[sloshburch]

Nope. I mean to ask how we delete our change (in local) and revert back to the on that came out of the box (in default).

You can't edit or delete conf files in a Search Head Cluster - it will render the Cluster out of sync. I tried to articulate that in the first post with this "Since we can't delete the edited version of the knowledge object from the UI, and we can't manually edit the conf file, what is the recommended way to address this?" I'll try to edit the post to be more clear.

[sjohnson_splunk]

Your right, you have to stop all of the SHC members, run those deletes then start the cluster.

[sloshburch]

Wouldn't that still break the RAFT sync for the SHC?

[jkat54]

You should be able to delete the edited version from the UI, and it will not delete the version sitting in default.

Another method might be to "blank out" the local version via editing using the UI.

In the backend this would "mash" together like this:

Default savedsearches.conf:
[awesomeSearch]
awesomeNewSettings
...
more new greatSTuff

Local savedsearches.conf:
[awesomeSearch]
...nothing here...

When those mash together at "read config time" you'd only get the "awesomeNewSettings".

But does this all work? I cant test right now, so it's purely theory.