Deployment Architecture

How to delete local changes to knowledge objects in a search head cluster?


Hey there!

I hope someone can give me hints on working with knowledge objects in a distributed environment. At the moment I am struggling with the following situation:

  • I use the Deployer to deploy a custom app to my Search Head Cluster. The app provides Dashboards and Alerts.
  • Some of the app's users have write permissions. When they change a Dashboard or Alert, the config will be saved to myapp/local/ on the SHC members.
  • At some point, I want to revert the users' changes (doesn't matter why).

So how do I easily and centrally delete all the data under myapp/local/ on the SHC members? I only came up with un-deploying and re-deploying the app from the Deployer, but this causes a rolling cluster restart, and I don't want that.

Kind regards,


This is causing headaches for us as well. I presently have a shell script that can run the same delete command across our 12-node search cluster to remove files which I cannot delete via the UI, Rest API or Deployer. I'm leaning towards having a non-clustered development/staging environment where changes can be made more easily, then using git or rysnc to push changes to the Deployer, and then onto the production cluster, and locking the production cluster to read-only where it cannot be edited via the UI.

0 Karma


Since you just want to delete their knowledge objects I would think you could login to one SH node and delete them through the gui. Replication should take care of the rest assuming you have defined the replication port on each SH.

0 Karma


Yes, this works fine when the delete button is there. The problem arises when confs/dashboards are pushed from the Deployer to the default folder. In this case we need to delete on the Deployer shcluster/apps folder and then run splunk apply shcluster-bundle to delete the content from the search cluster. This is very similar to how the Deployment server works. Not terrible, just need to keep track of whether the content came from the UI or the Deployer to know how to delete.

It gets more complicated when confs/dashboards were pushed via the Deployer and then a user goes in and edits the object via the UI. This is a very common scenario for apps downloaded from Splunkbase. In this case there are two versions of the object, one in default and one in local with the user updates and Splunk does not let you delete the local version via the UI when there is another version in the default folder.

Also, I've tried via the REST API and I see the attribute "removable" set to 0 as in False.

So ultimately you're left with the choice of doing everything via the UI or everything via the Deployer, or write scripts to clean up the mess until Splunk has better options via the UI.

0 Karma

Revered Legend

You can write a script to delete the myapp/local/ directory from SHC members. You'd still need to refresh the SHC members for your file system changes to take place.

0 Karma

Path Finder

splunk should provide a simpler way for this on their GUI.

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...