topic Re: What is the best way to remove local config in a Search Head Cluster? in Deployment Architecture

What is the best way to remove local config in a search head cluster?

sloshburch — Wed, 10 Aug 2022 14:23:56 GMT

Scenario:
Anyone using a SHC (Search Head Cluster) implements apps from the Deployer. The deployer collapses the local and default config directories into default and pushes the config to the SHC members.

After normal usage, some of the knowledge objects in the app have evolved (like a savedsearch or a macro has been modified).

Eventually a new version of the app comes out and it has a butt kickin' nice new version of that very knowledge object. So, I stage the new version of the app on the deployer and push it out.

Unfortunately, the local folder edit of the knowledge object still takes precedent and the sweet new version (sitting in the default directory on the SHC members) is ignored.

How do we eliminate our version of the config and revert back to the one in the local directory?

Since we can't delete the edited version of the knowledge object from the UI, and we can't manually edit the conf file, what is the recommended way to address this?

More detail:
If you use the Deployer to send the splunk_app_aws to your Search Head Cluster you'll then have a bunch of cool knowledge objects that you can edit. Let's pretend I want to edit aws-accesslog-sourcetype(1) to adjust it for my environment. Before the edit, this config lives ONLY in $SPLUNK_HOME/etc/apps/splunk_app_aws/default/macros.conf on the Search Head Cluster Members. I make the my change in the UI and the result is this:

Notice my new definition of blah but no way to delete or revert back. There is now a corresponding version of this macro on the Search Head Cluster Members in $SPLUNK_HOME/etc/apps/splunk_app_aws/local/macros.conf defined as blah.

Now let's pretend that after some time, I want to remove my change and go back to the version provided in $SPLUNK_HOME/etc/apps/splunk_app_aws/default/macros.conf - with a single search head OR a search head pool, I can simply remove the corresponding stanza in $SPLUNK_HOME/etc/apps/splunk_app_aws/local/macros.conf with a text editor and restart the instance thereby allowing the version in $SPLUNK_HOME/etc/apps/splunk_app_aws/default/macros.conf to take affect.

Unfortunately, you cannot make manual edits to configuration in a Search Head Cluster. So is there a parallel way to remove your $SPLUNK_HOME/etc/apps/splunk_app_aws/local/macros.conf version in a Search Head Cluster?

Re: What is the best way to remove local config in a Search Head Cluster?

jkat54 — Wed, 07 Dec 2016 19:09:00 GMT

You should be able to delete the edited version from the UI, and it will not delete the version sitting in default.

Another method might be to "blank out" the local version via editing using the UI.

In the backend this would "mash" together like this:

Default savedsearches.conf:
[awesomeSearch]
awesomeNewSettings
...
more new greatSTuff

Local savedsearches.conf:
[awesomeSearch]
...nothing here...

When those mash together at "read config time" you'd only get the "awesomeNewSettings".

But does this all work? I cant test right now, so it's purely theory.

Re: What is the best way to remove local config in a Search Head Cluster?

sjohnson_splunk — Wed, 07 Dec 2016 20:03:39 GMT

First, I assume you meant to say default here: How do we eliminate our version of the config and revert back to the one in the local directory?

If you want to remove all the local conf files here is a simple command line that will do it very quickly on a search head.

WARNING: backup of etc/apps recommended first!

splunk btool check --debug | egrep ^Checking.+/apps.+/local | sed -e 's/Checking: //' | xargs rm

You could add another grep in the list if you want to limit it to only certain apps.

Be careful, good luck!

Re: What is the best way to remove local config in a Search Head Cluster?

sloshburch — Thu, 08 Dec 2016 18:38:09 GMT

Nah, you can't delete from the UI. For example, you can edit a macro that comes from the AWS app and the UI doesn't provide a way to delete your edit to revert back to the default version.

Setting a value to blank in the UI would leave an entry in the conf file but with no value - effectively unsetting the parameter. This would still override the version in default with a value of nothing.

Re: What is the best way to remove local config in a Search Head Cluster?

sloshburch — Thu, 08 Dec 2016 18:40:19 GMT

Nope. I mean to ask how we delete our change (in local) and revert back to the on that came out of the box (in default).

You can't edit or delete conf files in a Search Head Cluster - it will render the Cluster out of sync. I tried to articulate that in the first post with this "Since we can't delete the edited version of the knowledge object from the UI, and we can't manually edit the conf file, what is the recommended way to address this?" I'll try to edit the post to be more clear.

Re: What is the best way to remove local config in a Search Head Cluster?

sjohnson_splunk — Thu, 08 Dec 2016 20:30:58 GMT

Your right, you have to stop all of the SHC members, run those deletes then start the cluster.

Re: What is the best way to remove local config in a Search Head Cluster?

sloshburch — Fri, 09 Dec 2016 13:38:51 GMT

Wouldn't that still break the RAFT sync for the SHC?

Re: What is the best way to remove local config in a Search Head Cluster?

sloshburch — Fri, 09 Dec 2016 13:39:28 GMT

This also doesn't work with a REST command:

$ curl -k --request DELETE https://host:port/servicesNS/nobody/splunk_app_aws/configs/conf-macros/aws-accesslog-sourcetype%281%29

<msg type="ERROR">Object id=aws-accesslog-sourcetype(1) cannot be deleted in config=macros.</msg>

Re: What is the best way to remove local config in a Search Head Cluster?

goodsellt — Fri, 09 Dec 2016 14:27:13 GMT

The ways I can think of would be to either:
1. Stop all SHC members, remove the local folder for the app, start them up again.
2. Move the app off the SHC deployer, let the SHC delete the app from themselves, redeploy.
3. I've never tried this, but maybe it could work: instead of using the SHC deployer, deploy everything to the SHC captain from a regular deployment server or manually and see if it'll replicate everything across the rest of the members anyways.

Re: What is the best way to remove local config in a Search Head Cluster?

sloshburch — Mon, 12 Dec 2016 13:13:07 GMT

Yea, but I don't think any of us are really satisfied with those, right?

I have a feeling this is likely to produce a sync issue with RAFT and would not really be supported
Def would work, but before you move it off the deployer, you'd have to throw a copy of the SHC member's local folder on the version to get re-deployed from the deployer, otherwise you'd lose all the Knowledge Objects folks created
This sounds like bad-news-bears and I'm sure any problems would not be supported.

Essentially, option 2 is the best but I've also submitted a feature request for this. Essentially, the non-SHC usage of splunk allowed this scenario to be addressed trivially and as such, no deliberate feature needed to be created to support it.

Re: What is the best way to remove local config in a Search Head Cluster?

goodsellt — Mon, 12 Dec 2016 14:04:23 GMT

Yea they're all bad, but it's a side effect of the odd design choice to have a "deployment" master server which then can start having radically different versions from the captain and the other SHC members. Good idea for a feature request, seems like something they need to address in one way or the other.

Re: What is the best way to remove local config in a Search Head Cluster?

alisterwhipp — Mon, 27 Mar 2017 23:12:45 GMT

If you edit a local config to be exactly the same as the deployed default, does that make the local go away?

Re: What is the best way to remove local config in a Search Head Cluster?

sloshburch — Tue, 28 Mar 2017 12:52:50 GMT

No. I talked with some Splunkers internally and learned that some Knowledge Objects have that behavior but not all. For many types of Knowledge Objects it merely persists a local config of the string you entered regardless of if it matches what was originally in default. 😞

Re: What is the best way to remove local config in a Search Head Cluster?

alisterwhipp — Tue, 28 Mar 2017 21:22:17 GMT

I did wonder. It would require exactly matching.
For dashboards what we've been doing is removing the dashboard on the deployment, deleting on the shc, then re-adding via deployer. This of course means that dashboard is actually missing in action for as long as it takes to re-deploy after deleting the local. And, I have no idea if this would work for anything stored in savedsearches.conf.

Re: What is the best way to remove local config in a Search Head Cluster?

gjanders — Wed, 29 Mar 2017 08:02:02 GMT

Thankyou for submitting the feature request, this is one of the most frustrating things relating to search head clustering. I've got quite a few enhancement requests open and I'm noticing the ones I'm finding on the forums are generally not implemented.

I've seen other companies ignore feature requests for years , I'm hoping Splunk is different!

Re: What is the best way to remove local config in a Search Head Cluster?

koshyk — Wed, 29 Mar 2017 10:40:37 GMT

I have asked atleast 5 questions on same issues. This default and local is such a hardwork on clusters and I hope Splunk will look into this

Re: What is the best way to remove local config in a Search Head Cluster?

koshyk — Wed, 29 Mar 2017 10:44:41 GMT

This is always a problem with Splunk whereby from Deployer it pushes to "default" in SH members. What we do is
- Strict Control for End Users by process and roles: They cannot create dashboards/config elements on Apps. But only on personal space. If anything to be deployed to App, it should come to Splunk Platform team and will be deployed via deployer
- In case if you have a local entry, the splunk engineer should merge this after carefully looking into source-control + deployer. Then sh-deploy and wait for it to finish. Then STOP all SH members , delete it from all at same time from local, clear raft. Restart and redeploy again from deployer to make it 100% sure.

But if you have this problem already, the only reliable workaround I've found is
- STOP all the search heads at the approximate time.
- Take backup of files in "local" directory. i.e. /opt/splunk/etc/apps//local/.conf
- Ensure all SEarch heads are stopped .( Yes, I know this is impacting.. but hey.. its SHC pain)
- Remove the local file itself from all search heads. Dont edit it.
- Start search heads and do a redeploy from deployer

Re: What is the best way to remove local config in a Search Head Cluster?

sloshburch — Fri, 31 Mar 2017 12:20:08 GMT

How do you delete on the SHC? If you do it on the filesystem you run the risk of a config mismatch...

Re: What is the best way to remove local config in a Search Head Cluster?

gn694 — Thu, 22 Mar 2018 14:25:57 GMT

I just tried this but it did not work for me:
1. Deleted dashbaord (xml file) on deployer
2. Applied the shcluster-bundle
3. Went to delete the dashboard on a Search Head but the delete option is still not available.
4. Verified on each SHCluster member in the app's default directory that the dashboard (xml file) is no longer present. It is now only in the local directory.
5. I restarted Splunk on one of the Search heads and still can not delete the (now local only) dashboard there.

Re: What is the best way to remove local config in a Search Head Cluster?

gn694 — Thu, 22 Mar 2018 18:54:52 GMT

Update: I forgot to also delete the details for the dashboard in the metadata file on the Deployer. Once that was also removed I was able to delete the local copy.