Alerting

8.0.4.1 on Ubuntu 20.04LTS: Why are emails failing with 'ResourceNotFound' error on alerts?

dkozinn
Path Finder
I'm running Splunk Enterprise 8.0.4.1 on Ubuntu 20.04LTS single user instance. I am using an Enterprise dev/test license (single user) for this instance. Any attempt to send email results in the following in python.log:

2020-07-07 21:45:15,136 +0000 ERROR     sendemail:1435 - [HTTP 404] https://127.0.0.1:8089/servicesNS/admin/search/saved/searches/_new?output_mode=json
Traceback (most recent call last):
File "/opt/splunk/etc/apps/search/bin/sendemail.py", line 1428, in <module>
results = sendEmail(results, settings, keywords, argvals)
File "/opt/splunk/etc/apps/search/bin/sendemail.py", line 261, in sendEmail
responseHeaders, responseBody = simpleRequest(uri, method='GET', getargs={'output_mode':'json'}, sessionKey=sessionKey)
File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 577, in simpleRequest
raise splunk.ResourceNotFound(uri)
ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/servicesNS/admin/search/saved/searches/_new?output_mode=json
The mailserver is on the local host, TLS isn't in use. I've verified that the Splunk user can send email both via the mail command as well as calling sendmail directly.  The error happens if I use an alert with email as well as using |sendemail from a search. 
 
As this instance is limited to a single user the user that this runs under has the admin role. For logging preferences, I set EmailSender to DEBUG but I'm not seeing anything useful.
Labels (2)
1 Solution

trashyroadz
Splunk Employee
Splunk Employee

I appreciate this discussion! Discovered that indeed the Dev/Test license only allows one user login, which is the main admin account. If that admin account is given a username other than 'admin', Splunk will not send alerts. If you cat the passwd file for your instance (cat /opt/splunk/etc/passwd), you'll see your main username listed with a hashed password, and the Administrator defined as 'admin', not as the main admin user.

Splunk alerts are sent from the 'admin' administrator account, which apparently will work on a non-dev/test license even if the admin account you set up as something other than 'admin'.

Solution: edit the passwd file to change the name of your user account to 'admin', then restart Splunk.

Please note, I also discovered if you delete the dev/test license and restart Splunk, Splunk will no longer recognized your admin account unless it is named 'admin'... in fact it will say there are no users for this deployment and won't allow you to log out, add/remove/modify users, etc. Again, the issue can be resolved by updating the passwd file and restarting Splunk. 

-- now that's Trashy!

View solution in original post

benhooper
Communicator

FYI: There seems to be no default username - you're required to enter one during the initial set-up.

2020 ∕ 10 ∕ 21 16꞉44꞉52 - root@ubuntuserver_~.png

Anyway, I set up the following new environment:

  • OS: Ubuntu Server 20.04
  • Splunk: Enterprise (full 2-month trial) 8.1.0
  • Username: admin
  • Alert owner: nobody
  • Email server: smtp.office365.com:587

I have not yet seen the aforementioned errors.

I re-added the logging which reported a URI of uri: /servicesNS/splunk-system-user/TA-<app name>/saved/searches/<alert name>

Why this is different and works, I'm not sure at this point but I'll look into it further tomorrow.

isoutamo
SplunkTrust
SplunkTrust
That’s true. Splunk removed default admin user at 7.2(?) and after that you should create it when you install splunk at first time.
@dkozinn can you still remember if you have admin user or not or reconstruct this?
r. Ismo
0 Karma

dkozinn
Path Finder

Thanks for digging into this @benhooper . I'd started digging into this and got stuck due to the same lack of documentation that you mentioned.

0 Karma

trashyroadz
Splunk Employee
Splunk Employee

I appreciate this discussion! Discovered that indeed the Dev/Test license only allows one user login, which is the main admin account. If that admin account is given a username other than 'admin', Splunk will not send alerts. If you cat the passwd file for your instance (cat /opt/splunk/etc/passwd), you'll see your main username listed with a hashed password, and the Administrator defined as 'admin', not as the main admin user.

Splunk alerts are sent from the 'admin' administrator account, which apparently will work on a non-dev/test license even if the admin account you set up as something other than 'admin'.

Solution: edit the passwd file to change the name of your user account to 'admin', then restart Splunk.

Please note, I also discovered if you delete the dev/test license and restart Splunk, Splunk will no longer recognized your admin account unless it is named 'admin'... in fact it will say there are no users for this deployment and won't allow you to log out, add/remove/modify users, etc. Again, the issue can be resolved by updating the passwd file and restarting Splunk. 

-- now that's Trashy!

dkozinn
Path Finder

Thank you so much for following up on this. I modified the username in passwd and finally alerts are working. As a bonus, I believe that may have fixed an issue with search history always being blank despite the etc/users/myuser/search/myhost.csv file being populated.  I hadn't checked that in a while so I can't be 100% sure this was the fix, but it is working now.

Rather than try to recreate saved searches, alerts, etc., can I copy everything under ./etc/users/olduser to ./etc/users/admin? I tried copying a few things but I suspect there are things I'm missing (stuff under various metadata directories?). I haven't done anything I care about after I modified the username so I don't care if anything there gets blown away.

trashyroadz
Splunk Employee
Splunk Employee

Glad to hear this solution helped, @dkozinn ! This thread seemed dated, but it sent me in the right direction for troubleshooting, so figured I'd post my findings anyway. It appears to occur across Splunk versions also.

I also noticed the disappearing search history, and it did seem to come back when the account was renamed to 'admin', at least up to the point where the issue started. I had tested both using searches with "sendemail" and setting up scheduled alerts. The scheduled alerts also came back online once the account was renamed 'admin'.

-- now that's Trashy!
0 Karma

dkozinn
Path Finder

To be honest @trashyroadz I was surprised that anyone found this and followed up after all this time, so it's much appreciated.

You might have missed the other half of my last message:

 Rather than try to recreate saved searches, alerts, etc., can I copy everything under ./etc/users/olduser to ./etc/users/admin? I tried copying a few things but I suspect there are things I'm missing (stuff under various metadata directories?). I haven't done anything I care about after I modified the username so I don't care if anything there gets blown away.

 

0 Karma

trashyroadz
Splunk Employee
Splunk Employee

Let me see what I can find out about that. Before updating the passwd file I had tried copying things over and was not getting any changes/results from doing so.

-- now that's Trashy!
0 Karma

dkozinn
Path Finder

I wish this was a VM as I'd just take a snapshot before trying this. Obviously it's easy enough to save and restore the files, but I don't know what happens internally in Splunk and as mentioned before, I'd rather not have to build the per-user stuff again from scratch if I can help it.

0 Karma

dkozinn
Path Finder

Following-up on my follow-up, I noticed a few errors when restarting splunk, and to make a long story short, found that I'd copied some things from the original user directory to the admin user directory as root but never changed the file ownership. As a result, it seems that Splunk wasn't able to read those files. Running 

chown -R splunk:splunk

from $SPLUNK_ETC/users/admin fixed that, and now I'm seeing the old saved searches, etc.

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...