topic Re: Issue with Users Unable to Delete Reports in Splunk SHC in Splunk Search

Issue with Users Unable to Delete Reports in Splunk SHC

m_zandinia — Sat, 17 May 2025 13:10:39 GMT

Hi Splunkers,

I’m running a Splunk Search Head Cluster (SHC) with 3 search heads, authenticated via Active Directory (AD). We have several custom apps deployed.

Currently, users are able to:

Create alerts
Delete alerts
Create reports
However, they are unable to delete reports.

Investigation Details

From the _internal logs, here’s what I observed:

When deleting an alert — the deletion works fine:

192.168.0.1 - user [17/May/2025:11:06:59.687 +0000] "DELETE /en-US/splunkd/__raw/servicesNS/username/SOC/saved/searches/test-user-alert?output_mode=json HTTP/1.1" 200 421 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:138.0) Gecko/20100101 Firefox/138.0" - eac203572253a2bd3db35ee0030c6a76 68ms 192.168.0.1 - user [17/May/2025:11:06:59.690 +0000] "DELETE /servicesNS/username/SOC/saved/searches/test-user-alert HTTP/1.1" 200 421 "-" "Splunk/9.4.1 (Linux 6.8.0-57-generic; arch=x86_64)" - - - 65ms

When deleting a report — it fails with a 404 Not Found:

192.168.0.1 - user [17/May/2025:10:27:51.699 +0000] "DELETE /en-US/splunkd/__raw/servicesNS/nobody/SOC/saved/searches/test-user-report?output_mode=json HTTP/1.1" 404 84 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:138.0) Gecko/20100101 Firefox/138.0" - eac203572253a2bd3db35ee0030c6a76 5ms 192.168.0.1 - user [17/May/2025:10:27:51.702 +0000] "DELETE /servicesNS/nobody/SOC/saved/searches/test-user-report HTTP/1.1" 404 84 "-" "Splunk/9.4.1 (Linux 6.8.0-57-generic; arch=x86_64)" - - - 1ms

Alerts are created under the user’s namespace (servicesNS/username/...) and can be deleted by the user.
Reports appear to be created under the nobody namespace (servicesNS/nobody/...), which may be the reason users lack permission to delete them.

Has anyone faced a similar issue?

Re: Issue with Users Unable to Delete Reports in Splunk SHC

kiran_panchavat — Sat, 17 May 2025 15:05:57 GMT

@m_zandinia

It means that the user that created the object is no longer a user in the authenticating system. If you create a local user, then login as that user, then create any knowledge object, then delete that user, then all of his KOs will switch to be owned by nobody.

What are "splunk-system-user" and nobody"? - Splunk Community

Solved: What does 'nobody' (under owner column) signify in... - Splunk Community

saved searches become orphaned or are associated with users that no longer exist. In such cases, reassigning the saved search to an existing valid user and then deleting it via the GUI can resolve the issue.

https://community.splunk.com/t5/Deployment-Architecture/Orphaned-Scheduled-Search-cannot-delete/m-p/221185

Re: Issue with Users Unable to Delete Reports in Splunk SHC

kiran_panchavat — Sat, 17 May 2025 15:26:01 GMT

@m_zandinia

Are they unable to delete all reports or just specific ones? Did you check the permissions for the reports?

Re: Issue with Users Unable to Delete Reports in Splunk SHC

kiran_panchavat — Sat, 17 May 2025 15:28:41 GMT

@m_zandinia

Check this

| rest /services/saved/searches 
| table title, eai:acl.owner, eai:acl.app, eai:acl.sharing, eai:acl.perms.read, eai:acl.perms.write
| rename title as "Report Name", 
eai:acl.owner as "Owner", 
eai:acl.app as "App", 
eai:acl.sharing as "Sharing Level", 
eai:acl.perms.read as "Read Permissions", 
eai:acl.perms.write as "Write Permissions"
| sort App, "Report Name"

Re: Issue with Users Unable to Delete Reports in Splunk SHC

livehybrid — Sat, 17 May 2025 15:43:42 GMT

Hi @m_zandinia

It sounds like in your scenario that your users can delete their own reports / knowledge objects within the app, but not owned by "nobody" (and probably not able to delete anything owned by any other user either). In order for them to delete App-level shared objects they will need to have write permission to the app - does their role have this? (Or the admin_all_objects capability)

Additionally if the knowledge object is globally shared they would need the admin_all_objects capability.

@kiran_panchavat Im not convinced those answers from 10 years ago are still valid, or only partially valid - If you delete a user which owns knowledge objects the owner does not get changed to "nobody". It stays owned by the original owner but becomes Orphaned:

In the below example I have 2 searches "owned" by "testing1" which I deleted, thus they become orphaned and still owned by the testing1 user.

Its not uncommon for things to be owned by "nobody" - whilst I prefer the use of service accounts, a lot of customers use the nobody user for owning artifacts which dont have a specific owner within an app (e.g. no specific named person, thus "nobody").

According to the docs an orphaned search "will not run the scheduled report on its schedule at all" - whereas a search owned by "nobody" will.

Its important to know that searches which are owned by "nobody" do get executed and will be essentially run as the system user, so will have access to all indexes, lookups etc, and therefore if someone has write access to a search owned by nobody that they could modify it to search in indexes which they themselves are now allowed to search! This is why I always recommend searches to be owned by a service account following the principals of least-privileged access.

🌟 Did this answer help you? If so, please consider:

Adding karma to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

Re: Issue with Users Unable to Delete Reports in Splunk SHC

m_zandinia — Sun, 18 May 2025 05:00:00 GMT

It's not an orphaned knowledge object.

A user creates a report and immediately afterward is unable to delete it.
A user creates an alert and can delete it immediately without issue.

The report is created in the correct location (as shown in the screenshot below).

However, the delete request sent by the Splunk UI is targeting the wrong URL, and I’m not sure why this is happening.

P.S. The admin is able to delete the report without any issue.
P.S. For privacy reasons, I manually changed the report name to "test-user-report" in the log samples. The actual report name matches what is shown in the screenshot.

Re: Issue with Users Unable to Delete Reports in Splunk SHC

kiran_panchavat — Sun, 18 May 2025 05:51:21 GMT

@m_zandinia

To delete a report in Splunk, a user must have the delete capability assigned to their role. This capability allows users to delete knowledge objects such as saved reports, alerts, and dashboards that they own or have permission to manage.

Create and edit reports - Splunk Documentation

By default admin has the capability called "admin_all_objects", so he can delete the report.

admin:

Created test user called "test-user" and assigned user & power user role to that, and can see that admin_all_objects has not assigned.

Re: Issue with Users Unable to Delete Reports in Splunk SHC

m_zandinia — Sun, 18 May 2025 07:22:25 GMT

Did you read my question carefully?

Users can delete their own alerts, but they cannot delete their own reports.

If they didn’t have the "delete_knowledge_objects" capability, how would they even be able to delete their own alerts in the first place?

This inconsistency is exactly the issue I’m trying to highlight.

Re: Issue with Users Unable to Delete Reports in Splunk SHC

isoutamo — Sun, 18 May 2025 12:29:04 GMT

There is no separate delete capability for reports. It works if you have modify/write capability, which you obviously have as you could create those reports under your private scope. That can_delete capability is totally different stuff and you never want to get it to normal user.

Re: Issue with Users Unable to Delete Reports in Splunk SHC

isoutamo — Sun, 18 May 2025 12:32:02 GMT

Have you check that your SHC is in sync and there is no general errors wit it or with kvstore?

I suppose that your users have created those alerts with gui and also they try to remove those via gui? Or are they trying to use REST api?

Re: Issue with Users Unable to Delete Reports in Splunk SHC

kiran_panchavat — Sun, 18 May 2025 13:11:13 GMT

@isoutamoYes, I understand that there isn't a specific capability called "delete." I was referring to the modify/write capability in a broader context.

Re: Issue with Users Unable to Delete Reports in Splunk SHC

m_zandinia — Sun, 18 May 2025 14:09:56 GMT

@isoutamo

Thanks.

The SHC is in the sync and there is no issue with kvstore.

Yes. They create both the alerts and reports from Web UI. They can delete alerts from the Web UI but they can't delete their own reports from the Web UI.

I'm using Splunk Enterprise Version 9.4.1.

Re: Issue with Users Unable to Delete Reports in Splunk SHC

kiran_panchavat — Sun, 18 May 2025 15:14:37 GMT

@m_zandinia I can confirm that this is a known issue in version 9.4.1 & your report is private report as per the above screenshot.

https://docs.splunk.com/Documentation/Splunk/9.4.1/ReleaseNotes/KnownIssues

It got fixed in 9.4.2

https://docs.splunk.com/Documentation/Splunk/9.4.2/ReleaseNotes/Fixedissues

Re: Issue with Users Unable to Delete Reports in Splunk SHC

kiran_panchavat — Sun, 18 May 2025 15:28:25 GMT

@m_zandinia

You can use REST API to DELETE the report.

I have created test report and deleted using the below.

curl -k -u admin:password --request DELETE https://<splunk_host>:8089/servicesNS/admin/search/saved/searches/testreport

NOTE: -k: Bypasses SSL certificate verification

Reference: https://<splunk_host>:8089/servicesNS/<owner>/<app>/saved/searches/<report_name>

Re: Issue with Users Unable to Delete Reports in Splunk SHC

isoutamo — Sun, 18 May 2025 16:30:35 GMT

Can you ask that they try to delete those by REST api if they have access to it?

Ok, @kiran_panchavat has found from release notes that this is known issue and it has fixed on 9.4.2 which has already published and which also fix some security issues known in earlier versions.

Re: Issue with Users Unable to Delete Reports in Splunk SHC

m_zandinia — Mon, 19 May 2025 05:09:14 GMT

Thanks @kiran_panchavat

I did read that page, but probably not carefully enough the first time.

After reviewing the known issue more closely, I tested a potential workaround:
If the user changes the report’s permissions to allow read access for everyone, they are then able to delete the report.

According to the known issue, users cannot delete private reports. So I thought maybe they can delete public reports—and that turned out to be correct in my case.

Re: Issue with Users Unable to Delete Reports in Splunk SHC

kiran_panchavat — Mon, 19 May 2025 05:42:03 GMT

@m_zandinia

Great. It’s good to know that changing the report’s permissions fixed the issue.