Yes, permissions can get pretty messy. This also happens if Splunk is normally run by the Splunk user but is then started from root (usually using sudo). ... View more

Agreed! I had reproduced the issue again on another instance and wanted to document the steps to changing ownership. It's a little different depending on whether the original search was shared or not. Here's a recap of my findings: Alerts on Dev/Test licenses do work if the account for the instance IF: The login account is named 'admin' (the Dev/Test license only allows one user account, and it must be named 'admin'); AND In Advance Settings for the alert, the server is changed from 'localhost' to your organization's email server (for me, mail.splunk.com). To change the name of the login account to 'admin', update the passwd file to reference ":admin:" at the start of it instead of your original username (example: ":choppedliver:"), then restart Splunk. To copy the search history and saved searches / alerts over to the admin account: copy the history and local directories from $SPLUNK_HOME/etc/users/<originalusername> over to $SPLUNK_HOME/etc/users/admin. You can delete them from the <originalusername> folder also if you wish. NOTE: this instruction assumes your admin account is fresh and doesn't already have a history or local directory. If you have been using the admin account, decide which search history you want to keep. Combining them doesn't seem to work. If you already created saved searches for the admin account, copy the contents of the local/savedsearches.conf file from your original user over to the savedsearches.conf file currently existing for your admin account. For shared alerts, update the local.meta file to assign shared alerts from <originalusername> to admin ($SPLUNK_HOME/etc/apps/search/metadata/local.meta) Once again, restart Splunk for these changes to go into effect. ... View more

Let me see what I can find out about that. Before updating the passwd file I had tried copying things over and was not getting any changes/results from doing so. ... View more

Glad to hear this solution helped, @dkozinn ! This thread seemed dated, but it sent me in the right direction for troubleshooting, so figured I'd post my findings anyway. It appears to occur across Splunk versions also. I also noticed the disappearing search history, and it did seem to come back when the account was renamed to 'admin', at least up to the point where the issue started. I had tested both using searches with "sendemail" and setting up scheduled alerts. The scheduled alerts also came back online once the account was renamed 'admin'. ... View more