Is this documented anywhere?
I noticed that one of my scripted alerts stopped working once the owner account was deleted. The script and saved search were still there, but the saved search would not run. Once I changed ownership to another account everything started to function as expected.
What is the best practice to that maintain knowledge objects alive even if owner account is no longer present?
In the case of alerts and scheduled saved searches, they would not work since they run as the creator. If the user designated as creator no longer exists, the searches simply won't run. These end up what is known as "Orphaned Searches"
As a best practice, to avoid this scenario when a user is no longer with the company(or gets hit by a bus), production alerts and saved searches should be re-assigned to a service credential which does not expire. Service credentials are generic credentials which are not assigned to a particular user. This also works well when you have to restrict number of user searches without affecting the scheduled searches/alerts.
We put every
Knowledge Object inside of a custom app and then make sure that the
local.meta file is continuously stripped of ownership details which means that everything is owned by the
non-user known as
nobody. The only other option is to create a local system account in Splunk and ensure that everything is owned by that user.
The objects are stored in $splunkhome/users/$username/ with saved searches in $splunkhome/users/$username/search/local/savedsearches.conf specifically. Based on the testing I just did, deleting a user no longer removes those files, so you may be able to go grab copies of that user tree. There may be some housekeeping done at some point to remove those, but you can go look. Simply copy the objects you want out of those files (make a backup of the tree somewhere) into the user you want to own it or at the system level and restart Splunk.
no the, objects are not deleted, but they lose all the permissions associated.
Therefore if the scheduled search as the user A, then it will stop to run because there is no role associated since A is deleted. (and no role linked to).
So this is expected that the scheduled searches will try and fail because of permissions.
In recent versions of Splunk Enterprise (at least from 7.1), there is a pre-built dashboard in "Search and Reporting" app named "Orphaned Scheduled Searches, Reports, and Alerts". This shows the saved scheduled searches, reports, and alerts with deactivated owners. If you want to schedule it, simply copy the SPL powering the dashboard and schedule it. In my Splunk, here is the search powering that dashboard.
| rest timeout=600 splunk_server=local /servicesNS/-/-/saved/searches add_orphan_field=yes count=0
| search orphan=1 disabled=0 is_scheduled=1
| eval status = if(disabled = 0, "enabled", "disabled")
| fields title eai:acl.owner eai:acl.app eai:acl.sharing orphan status is_scheduled cron_schedule next_scheduled_time next_scheduled_time actions
| rename title AS "search name" eai:acl.owner AS owner eai:acl.app AS app eai:acl.sharing AS sharing
So the interesting part for me is that when the user gets deleted the scheduled are no longer able to run even though they are there. I would like for Splunk to leave artifacts in place and let them run at their configured scheduled.