8.0.4.1 on Ubuntu 20.04LTS: Why are emails failing with 'ResourceNotFound' error on alerts?

[dkozinn]

I'm running Splunk Enterprise 8.0.4.1 on Ubuntu 20.04LTS single user instance. I am using an Enterprise dev/test license (single user) for this instance. Any attempt to send email results in the following in python.log:

2020-07-07 21:45:15,136 +0000 ERROR     sendemail:1435 - [HTTP 404] https://127.0.0.1:8089/servicesNS/admin/search/saved/searches/_new?output_mode=json
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/search/bin/sendemail.py", line 1428, in <module>
    results = sendEmail(results, settings, keywords, argvals)
  File "/opt/splunk/etc/apps/search/bin/sendemail.py", line 261, in sendEmail
    responseHeaders, responseBody = simpleRequest(uri, method='GET', getargs={'output_mode':'json'}, sessionKey=sessionKey)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 577, in simpleRequest
    raise splunk.ResourceNotFound(uri)
ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/servicesNS/admin/search/saved/searches/_new?output_mode=json 

The mailserver is on the local host, TLS isn't in use. I've verified that the Splunk user can send email both via the mail command as well as calling sendmail directly.  The error happens if I use an alert with email as well as using |sendemail from a search. 

As this instance is limited to a single user the user that this runs under has the admin role. For logging preferences, I set EmailSender to DEBUG but I'm not seeing anything useful.

[benhooper]

FYI: There seems to be no default username - you're required to enter one during the initial set-up.

Anyway, I set up the following new environment:

• OS: Ubuntu Server 20.04
• Splunk: Enterprise (full 2-month trial) 8.1.0
• Username: admin
• Alert owner: nobody
• Email server: smtp.office365.com:587

I have not yet seen the aforementioned errors.

I re-added the logging which reported a URI of uri: /servicesNS/splunk-system-user/TA-<app name>/saved/searches/<alert name>

Why this is different and works, I'm not sure at this point but I'll look into it further tomorrow.

[isoutamo]

That’s true. Splunk removed default admin user at 7.2(?) and after that you should create it when you install splunk at first time.
@dkozinn can you still remember if you have admin user or not or reconstruct this?
r. Ismo

[dkozinn]

Thanks for digging into this @benhooper . I'd started digging into this and got stuck due to the same lack of documentation that you mentioned.

[trashyroadz · Accepted solution]

I appreciate this discussion! Discovered that indeed the Dev/Test license only allows one user login, which is the main admin account. If that admin account is given a username other than 'admin', Splunk will not send alerts. If you cat the passwd file for your instance (cat /opt/splunk/etc/passwd), you'll see your main username listed with a hashed password, and the Administrator defined as 'admin', not as the main admin user.

Splunk alerts are sent from the 'admin' administrator account, which apparently will work on a non-dev/test license even if the admin account you set up as something other than 'admin'.

Solution: edit the passwd file to change the name of your user account to 'admin', then restart Splunk.

Please note, I also discovered if you delete the dev/test license and restart Splunk, Splunk will no longer recognized your admin account unless it is named 'admin'... in fact it will say there are no users for this deployment and won't allow you to log out, add/remove/modify users, etc. Again, the issue can be resolved by updating the passwd file and restarting Splunk. 

[dkozinn]

Thank you so much for following up on this. I modified the username in passwd and finally alerts are working. As a bonus, I believe that may have fixed an issue with search history always being blank despite the etc/users/myuser/search/myhost.csv file being populated.  I hadn't checked that in a while so I can't be 100% sure this was the fix, but it is working now.

Rather than try to recreate saved searches, alerts, etc., can I copy everything under ./etc/users/olduser to ./etc/users/admin? I tried copying a few things but I suspect there are things I'm missing (stuff under various metadata directories?). I haven't done anything I care about after I modified the username so I don't care if anything there gets blown away.

[trashyroadz]

Glad to hear this solution helped, @dkozinn ! This thread seemed dated, but it sent me in the right direction for troubleshooting, so figured I'd post my findings anyway. It appears to occur across Splunk versions also.

I also noticed the disappearing search history, and it did seem to come back when the account was renamed to 'admin', at least up to the point where the issue started. I had tested both using searches with "sendemail" and setting up scheduled alerts. The scheduled alerts also came back online once the account was renamed 'admin'.

[dkozinn]

To be honest @trashyroadz I was surprised that anyone found this and followed up after all this time, so it's much appreciated.

You might have missed the other half of my last message:

 Rather than try to recreate saved searches, alerts, etc., can I copy everything under ./etc/users/olduser to ./etc/users/admin? I tried copying a few things but I suspect there are things I'm missing (stuff under various metadata directories?). I haven't done anything I care about after I modified the username so I don't care if anything there gets blown away.

[trashyroadz]

Let me see what I can find out about that. Before updating the passwd file I had tried copying things over and was not getting any changes/results from doing so.

[dkozinn]

I wish this was a VM as I'd just take a snapshot before trying this. Obviously it's easy enough to save and restore the files, but I don't know what happens internally in Splunk and as mentioned before, I'd rather not have to build the per-user stuff again from scratch if I can help it.

[dkozinn]

Following-up on my follow-up, I noticed a few errors when restarting splunk, and to make a long story short, found that I'd copied some things from the original user directory to the admin user directory as root but never changed the file ownership. As a result, it seems that Splunk wasn't able to read those files. Running 

chown -R splunk:splunk

from $SPLUNK_ETC/users/admin fixed that, and now I'm seeing the old saved searches, etc.