I've tried this as a solution without any success. ... View more

My understanding is that Splunk 8 still uses Python 2 for some features, including those involved in this error (specifically, alert actions). https://docs.splunk.com/Documentation/Splunk/8.0.5/Python3Migration/ChangesEnterprise That said, I'm no Splunk developer and might be interpreting that incorrectly. In the interest of troubleshooting I did change the default interpreter from 2.7 to 3.7 by setting "python.version = python3" in /opt/splunk/etc/system/local/server.conf and restarting Splunk. Re-running the same command resulted in nearly the same error, only with python3.7 being referenced.   2020-08-01 08:51:01,165 -0400 ERROR sendemail:1454 - [HTTP 404] https://127.0.0.1:8089/servicesNS/admin/search/saved/searches/_new?output_mode=json Traceback (most recent call last): File "/opt/splunk/etc/apps/search/bin/sendemail.py", line 1447, in <module> results = sendEmail(results, settings, keywords, argvals) File "/opt/splunk/etc/apps/search/bin/sendemail.py", line 262, in sendEmail responseHeaders, responseBody = simpleRequest(uri, method='GET', getargs={'output_mode':'json'}, sessionKey=sessionKey) File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 577, in simpleRequest raise splunk.ResourceNotFound(uri) splunk.ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/servicesNS/admin/search/saved/searches/_new?output_mode=json   I do see that it's looking for a results file that is not available. However, I don't understand how this could occur or where to start looking for the issue. If I had to make a guess, my gut feeling is that it's a Linux file permissions issue somewhere. Splunk is trying to create a file in a location that it does not have write access to. The file is never created, and a subsequent attempt to read the file fails. I have absolutely nothing to base that on though, it's just the only reason I can think of that Splunk is unable to find a file that it should be creating. ... View more

I don't know if you've figured this out yet, but I'm running into the exact same problem, and your thread seems to be the only one with the exact same error message. I'm also using a dev/test license, however, I'm trying to send through a third party mail provider (I tried two in fact, in case that was the issue). I'm running Splunk 8.0.5 on Ubuntu 18.04.4     2020-07-31 22:38:49,185 -0400 ERROR sendemail:1454 - [HTTP 404] https://127.0.0.1:8089/servicesNS/admin/search/saved/searches/_new?output_mode=json Traceback (most recent call last): File "/opt/splunk/etc/apps/search/bin/sendemail.py", line 1447, in <module> results = sendEmail(results, settings, keywords, argvals) File "/opt/splunk/etc/apps/search/bin/sendemail.py", line 262, in sendEmail responseHeaders, responseBody = simpleRequest(uri, method='GET', getargs={'output_mode':'json'}, sessionKey=sessionKey) File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 577, in simpleRequest raise splunk.ResourceNotFound(uri) ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/servicesNS/admin/search/saved/searches/_new?output_mode=json     ... View more

That's actually exactly what's in place. However, the internal log format is always timestamped with the #epoch timestamp. The behavior is described here: https://unix.stackexchange.com/questions/214322/write-bash-history-to-a-file-with-a-timestamp In other words, if you can the raw log, regardless of HISTTIMEFORMAT, you get #. Since Splunk is reading the raw log is what it gets. ... View more

No luck, it appears to be line breaking at the correct place, as my original props.conf did. However, it's still not parsing the timestamp. ... View more

No luck, it's breaking... weird. So one event comes in as hi this is a text #1579273320 exit And the previous one as: 1579273315 (the timestamp minux the #). It appears to alternate like this. Neither appears to be actually using this as the timestamp for the event though. ... View more

It was deployed from the deployment server within the Splunk_TA_nix app to the UF's (so /opt/splunk/etc/deployment-apps/Splunk_TA_nix/local/) ... View more

After looking in a few logs where I would expect and error to be (if there was one) I did a grep of -all- logs in /opt/splunk/var/log/splunk/ for "bash" and found nothing. Is there a specific log and/or keyword you know to check for? ... View more

@scottrunyon - Can you post your full \local\props.conf and \default\props.conf ? We have the same problem, but when we unset indexed_extractions and kv_mode, we just get a giant chunk of raw json. I'm not sure where else our settings differ. ... View more