sendemail stopped working all of a sudden

[sylim_splunk]

Sendemail apparently stopped working and python.log  has the log messages as below repeating whenever I test "sendemail" or email alert was triggered. But then the email appears to be discarded silently. The mail server doesn't have any logs from the SH which means, according to Email admin, there's no connection attempted from the SH.

Symptoms: no accounds including admin are able to send email alert.

Log message in python.log

2020-08-05 09:10:42,196 -0700 ERROR     sendemail:1421 - Only absolute URIs are allowed. uri = No mapping for safe name: D:\Program Files\Splunk\etc\users\users.ini: pac_adm.m.11905a0d206502752431c8c204542692^M
No mapping for safe name: D:\Program Files\Splunk\etc\users\users.ini: pac_adm.m.11905a0d206502752431c8c204542692^M
<SNIP>
No mapping for safe name: D:\Program Files\Splunk\etc\users\users.ini: pac_vik.b.69565216b720a1546ffc9ae8d5aa120b^M
No mapping for safe name: D:\Program Files\Splunk\etc\users\users.ini: pac_wil.p.1294be4f866029819ac22bdc54e1695b^M
No mapping for safe name: D:\Program Files\Splunk\etc\users\users.ini: pac_xao.c.6f1004219c3b1ceb9a5cee5f5dea227d^M
https://127.0.0.1:8089/servicesNS/admin/search/saved/searches/_new?output_mode=json

[sylim_splunk · Accepted solution]

The file, users.ini is there to map the user account and its directory in $SPLUNK_HOME/etc/users - it is created if the file system is case insensitve, has upper chars in user account or user account name has characters that are not allowed as a file name. Splunk creates _reserved and users.ini to map the user account and the directory. The file, users.ini is dynamically updated over time - whenever users logs in to Search heads  the entry gets populated as well as the user directory under _reserved. But it's not replicated accross the SHCluster.

* Why then it just stopped working with the messages?

Even if it's changing the contents dynamically it is also tracked by the "File Integrity Checker" by mistake. - Ver8.0.1+ have excluded the users.ini from the integrity checker.  For the versions prior to 8.0.1, Integrity Checker warns users with "it did not match the system-provided manifest. Review the list of problems reported by the Installed FileHashChecker in splunkd.log..."

It is suspected that the users.ini might have been truncated or restored in the production as reportely some admins decided to truncate the file due to the annoying message from the integrity checker. Then the mismatch between _reserved/userDirectories and the content of users.ini happens.

* Why this is a problem?

Then the symptom of no email alerting happens - Splunk sendemail will stop working any more due to the mismatch of how it handles errors between splunkd and the open source lib around users.ini. As you have already noticed in the log message -

uri = No mapping .... https://127.0.0.1:8089/servicesNS/admin/search/saved/searches/_new?output_mode=json

The error in a library broke the uri that sendemail needs.

* Fix version: 7.3.8+ , 8.0.7+

* Workaround before upgrade:

Update the users.ini with the user information in etc/users/_reserved.

1. cd _reserved and run the script below to create list of users not found in users.ini.

for i in `ls -1` 
do 
echo "== reserved : $i"
user=$( grep "$i" ../users.ini |cut -d"=" -f 2 ) 
if [ "X$user" = "X" ]; then echo $i >> /tmp/notfound 
echo "NOT FOUND $i"
continue 
fi 
echo "FOUND $i = $user" 
done

2. Based on the notfound users add them into users.ini.

1858376