Can't Delete Dashboards/Reports/Indexes After Running Packaging Tool

[edgarrity]

We have Splunk Enterprise 8.1.2 and are preparing our app to migrate to a Splunk Cloud environment. After running the Splunk packaging tool to package our app for certification (Splunk package app), we noticed that the dashboards, reports, and indexes for our app can no longer be deleted from the Splunk Web UI or CLI. The delete button is missing for the dashboards and reports. The button is greyed out for the indexes. Running the CLI remove command returns a successful message but does not delete the object. Does anyone know how to re-enable the delete function so we can delete old knowledge objects we no longer need?

[Ed11375]

Splunk seems to have a hidden flag that is set when the Packaging Tool is ran on an app which causes Splunk Enterprise Web UI to remove the ability to delete any knowledge objects in the app. After I moved the indexes to another app, the delete function for the indexes was re-enabled in Splunk Enterprise. Since an app with indexes defined cannot be packaged for installation in Splunk Cloud, this makes sense. However, still do not understand why the dashboards and reports in an app cannot be deleted in Splunk Enterprise after running the Packaging Tool on the app. What is interesting is that when I restored a backup of the app made before the Packaging Tool was ran on it, the restored app did allow the dashboards and reports to be deleted in the Splunk Enterprise Web UI. So it seems the Splunk Packaging Tool does modify the app so that Splunk Enterprise Web UI cannot delete knowledge objects. Next I will compare the files in the backup copy to the files in the app after the Packaging Tool was run to see what Splunk changes in the app to cause Splunk Enterprise Web UI to disable the delete function on the dashboards and reports.

[Ed11375 · Accepted solution]

I found the problem. Comparing the files in the backup copy of the app to the files in the app after running through the Packaging Tool process, and systematically undoing any changes between the two versions, I found that the culprit was the local.meta file in the $SPLUNK_HOME\etc\apps\app\metadata folder. One of the recommendations made by the Packaging Tool was to merge the contents of the local.meta file with the default.meta file, then delete the local.meta file from the app. After restoring the local.meta file to the app, the delete menu item for the dashboards and reports for the app was restored in the Splunk Enterprise Web UI. I suggest that anyone who runs into this issue when preparing an app for deployment to Splunk Cloud, restore this file after running through the Packaging Tool process, so that development of the app can continue on your Splunk Enterprise System.

[edgarrity]

Splunk Support said they would be calling me today to review the issue.  I will post the outcome of the meeting in this ticket.  I am curious why deleting the local.meta file, as required by the “Vet apps and add-ons for Splunk Cloud” process in the Splunk documentation (https://dev.splunk.com/enterprise/docs/releaseapps/cloudvetting/) causes Splunk Enterprise to disable the delete function.

[edgarrity]

This morning, Splunk Support said they would be sending me a meeting link to discuss this bug.  The meeting was supposed to take place at 3PM ET.  That time came and went and support never sent the link. 

We need to move on with our app development to get it ready to migrate from Splunk Enterprise to Splunk Cloud.  So I will incorporate the work around into our deployment process and wait for support to let us know when the bug will be fixed.