Issue with Users Unable to Delete Reports in Splunk SHC

[m_zandinia]

Hi Splunkers,

I’m running a Splunk Search Head Cluster (SHC) with 3 search heads, authenticated via Active Directory (AD). We have several custom apps deployed.

Currently, users are able to:

• Create alerts
• Delete alerts
• Create reports
  However, they are unable to delete reports.

Investigation Details

From the _internal logs, here’s what I observed:

When deleting an alert — the deletion works fine:

192.168.0.1 - user [17/May/2025:11:06:59.687 +0000] "DELETE /en-US/splunkd/__raw/servicesNS/username/SOC/saved/searches/test-user-alert?output_mode=json HTTP/1.1" 200 421 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:138.0) Gecko/20100101 Firefox/138.0" - eac203572253a2bd3db35ee0030c6a76 68ms

192.168.0.1 - user [17/May/2025:11:06:59.690 +0000] "DELETE /servicesNS/username/SOC/saved/searches/test-user-alert HTTP/1.1" 200 421 "-" "Splunk/9.4.1 (Linux 6.8.0-57-generic; arch=x86_64)" - - - 65ms

When deleting a report — it fails with a 404 Not Found:

192.168.0.1 - user [17/May/2025:10:27:51.699 +0000] "DELETE /en-US/splunkd/__raw/servicesNS/nobody/SOC/saved/searches/test-user-report?output_mode=json HTTP/1.1" 404 84 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:138.0) Gecko/20100101 Firefox/138.0" - eac203572253a2bd3db35ee0030c6a76 5ms

192.168.0.1 - user [17/May/2025:10:27:51.702 +0000] "DELETE /servicesNS/nobody/SOC/saved/searches/test-user-report HTTP/1.1" 404 84 "-" "Splunk/9.4.1 (Linux 6.8.0-57-generic; arch=x86_64)" - - - 1ms

• Alerts are created under the user’s namespace (servicesNS/username/...) and can be deleted by the user.
• Reports appear to be created under the nobody namespace (servicesNS/nobody/...), which may be the reason users lack permission to delete them.

Has anyone faced a similar issue?

[m_zandinia]

It's not an orphaned knowledge object.

• A user creates a report and immediately afterward is unable to delete it.
• A user creates an alert and can delete it immediately without issue.

The report is created in the correct location (as shown in the screenshot below).

However, the delete request sent by the Splunk UI is targeting the wrong URL, and I’m not sure why this is happening.

P.S. The admin is able to delete the report without any issue.
P.S. For privacy reasons, I manually changed the report name to "test-user-report" in the log samples. The actual report name matches what is shown in the screenshot.

[kiran_panchavat]

@m_zandinia 

To delete a report in Splunk, a user must have the delete capability assigned to their role. This capability allows users to delete knowledge objects such as saved reports, alerts, and dashboards that they own or have permission to manage.

Create and edit reports - Splunk Documentation

By default admin has the capability called "admin_all_objects", so he can delete the report. 

admin:

Created test user called "test-user" and assigned user & power user role to that, and can see that admin_all_objects has not assigned. 

[isoutamo]

There is no separate delete capability for reports. It works if you have modify/write capability, which you obviously have as you could create those reports under your private scope. That can_delete capability is totally different stuff and you never want to get it to normal user.

[kiran_panchavat]

@isoutamoYes, I understand that there isn't a specific capability called "delete." I was referring to the modify/write capability in a broader context.

[m_zandinia]

Did you read my question carefully?

Users can delete their own alerts, but they cannot delete their own reports.

If they didn’t have the "delete_knowledge_objects" capability, how would they even be able to delete their own alerts in the first place?

This inconsistency is exactly the issue I’m trying to highlight.

[isoutamo]

Have you check that your SHC is in sync and there is no general errors wit it or with kvstore?

I suppose that your users have created those alerts with gui and also they try to remove those via gui? Or are they trying to use REST api?

[m_zandinia]

@isoutamo 

Thanks.

The SHC is in the sync and there is no issue with kvstore.

Yes. They create both the alerts and reports from Web UI. They can delete alerts from the Web UI but they can't delete their own reports from the Web UI.

I'm using Splunk Enterprise Version 9.4.1.

[isoutamo]

Can you ask that they try to delete those by REST api if they have access to it?

Ok, @kiran_panchavat has found from release notes that this is known issue and it has fixed on 9.4.2 which has already published and which also fix some security issues known in earlier versions.

[kiran_panchavat · Accepted solution]

@m_zandinia I can confirm that this is a known issue in version 9.4.1 & your report is private report as per the above screenshot. 

 https://docs.splunk.com/Documentation/Splunk/9.4.1/ReleaseNotes/KnownIssues 

It got fixed in 9.4.2

https://docs.splunk.com/Documentation/Splunk/9.4.2/ReleaseNotes/Fixedissues 

[m_zandinia]

Thanks @kiran_panchavat 

I did read that page, but probably not carefully enough the first time.

After reviewing the known issue more closely, I tested a potential workaround:
If the user changes the report’s permissions to allow read access for everyone, they are then able to delete the report.

According to the known issue, users cannot delete private reports. So I thought maybe they can delete public reports—and that turned out to be correct in my case.

[kiran_panchavat]

@m_zandinia 

Great. It’s good to know that changing the report’s permissions fixed the issue. 

[kiran_panchavat]

@m_zandinia 

You can use REST API to DELETE the report. 

I have created test report and deleted using the below. 

curl -k -u admin:password --request DELETE https://<splunk_host>:8089/servicesNS/admin/search/saved/searches/testreport

NOTE: -k: Bypasses SSL certificate verification

Reference: https://<splunk_host>:8089/servicesNS/<owner>/<app>/saved/searches/<report_name> 

[livehybrid]

Hi @m_zandinia 

It sounds like in your scenario that your users can delete their own reports / knowledge objects within the app, but not owned by "nobody" (and probably not able to delete anything owned by any other user either). In order for them to delete App-level shared objects they will need to have write permission to the app - does their role have this? (Or the admin_all_objects capability)

Additionally if the knowledge object is globally shared they would need the admin_all_objects capability.

@kiran_panchavat Im not convinced those answers from 10 years ago are still valid, or only partially valid - If you delete a user which owns knowledge objects the owner does not get changed to "nobody". It stays owned by the original owner but becomes Orphaned:

In the below example I have 2 searches "owned" by "testing1" which I deleted, thus they become orphaned and still owned by the testing1 user.

Its not uncommon for things to be owned by "nobody" - whilst I prefer the use of service accounts, a lot of customers use the nobody user for owning artifacts which dont have a specific owner within an app (e.g. no specific named person, thus "nobody").

According to the docs an orphaned search "will not run the scheduled report on its schedule at all" - whereas a search owned by "nobody" will.

Its important to know that searches which are owned by "nobody" do get executed and will be essentially run as the system user, so will have access to all indexes, lookups etc, and therefore if someone has write access to a search owned by nobody that they could modify it to search in indexes which they themselves are now allowed to search! This is why I always recommend searches to be owned by a service account following the principals of least-privileged access.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[kiran_panchavat]

@m_zandinia 

Check this 

| rest /services/saved/searches 
| table title, eai:acl.owner, eai:acl.app, eai:acl.sharing, eai:acl.perms.read, eai:acl.perms.write
| rename title as "Report Name", 
eai:acl.owner as "Owner", 
eai:acl.app as "App", 
eai:acl.sharing as "Sharing Level", 
eai:acl.perms.read as "Read Permissions", 
eai:acl.perms.write as "Write Permissions"
| sort App, "Report Name"

[kiran_panchavat]

@m_zandinia 

Are they unable to delete all reports or just specific ones? Did you check the permissions for the reports? 

[kiran_panchavat]

@m_zandinia 

It means that the user that created the object is no longer a user in the authenticating system. If you create a local user, then login as that user, then create any knowledge object, then delete that user, then all of his KOs will switch to be owned by nobody.

What are "splunk-system-user" and nobody"? - Splunk Community

Solved: What does 'nobody' (under owner column) signify in... - Splunk Community

saved searches become orphaned or are associated with users that no longer exist. In such cases, reassigning the saved search to an existing valid user and then deleting it via the GUI can resolve the issue.

https://community.splunk.com/t5/Deployment-Architecture/Orphaned-Scheduled-Search-cannot-delete/m-p/...