Reporting

Why does the sendemail command fail and display "('system library', 'fopen', 'No such file or directory')" error?

Builder

Hello!

We've been using Splunk here for quite some time (back since the 4.x days) but only recently have we had a need to use the "sendemail" command. The server can send email just fine when scheduling an alert and can do it just fine from the Linux command line. However, when we try to use the sendemail command by copying examples in the doc, we always get

command="sendemail", [('system library', 'fopen', 'No such file or directory'), ('BIO routines', 'BIO_new_file', 'no such file'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')] while sending mail to: me@mycompany.com

when I look in python.log I see pretty much the same non-informative information:

2016-10-19 16:33:35,866 -0400 INFO      sendemail:1067 - sendemail pdfgen_available = 1
2016-10-19 16:33:35,892 -0400 INFO      sendemail:1195 - sendemail:mail effectiveTime=None
2016-10-19 16:33:40,594 -0400 INFO      sendemail:1215 - Generated PDF for email
2016-10-19 16:33:40,658 -0400 ERROR     sendemail:131 - Sending email. subject="Here is an email notification 2", results_link="None", recipients="[u'me@mycompany.com']", server="internalmailhub.com"
2016-10-19 16:33:40,659 -0400 ERROR     sendemail:417 - [('system library', 'fopen', 'No such file or directory'), ('BIO routines', 'BIO_new_file', 'no such file'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')] while sending mail to: me@mycompany.com

So this seems like some kind of a python error with sendemail.py but it's not informative enough to clue me in to what might be wrong. Note that I've tried both using the "server" argument to sendemail and without it as both email types would work from this server (i.e. sending directly to a mail hub or letting it trickle through the internal mail system which would also punt the message to our internal mail hub).

Other useful information
- all servers are SLES 11.x
- default Linux mail configuration on our servers is for postfix. I believe it includes its own flavor of the "sendmail" command that is a work-alike to commands that want to call sendmail (i.e. "sendmail -t ..."). Don't know if that's a factor here
- servers we've tested this on are all search heads and we get consistent responses on each
- search heads are running 6.5.0

It probably goes without saying that if I try to just run the sendemail.py python script just to see if there's something obvious it can't find, it fails

$ python sendemail.py
Traceback (most recent call last):
  File "sendemail.py", line 1, in <module>
    import re, time, splunk.Intersplunk, splunk.mining.dcutils as dcu
ImportError: No module named splunk.Intersplunk

but I assume that's expected as it's not expecting to be run from an external python and rather from inside Splunk.

Thanks for your help!

-Mark

1 Solution

Builder

the X509loadcertcrlfile method/function seems like it's inside libcrypto and isn't any exposed python library. We don't have any CRL file.

I remember a long time ago having weird problems with either inputs.conf, server.conf or both when I used relative paths as the documentation suggested. Or maybe it was using $SPLUNK_HOME rather than relative paths. I can't remember. Either way it was unexpected at the time.

Out of curiosity, I was wondering if perhaps Splunk is passing $SPLUNKHOME/ directly into the libcrypto routines and not expanding that variable first. So I changed both the inputs.conf and server.conf SSL entries to not use $SPLUNKHOME but explicitly list the path out. Then I restarted. Then I stopped having the problem.

It's late and I don't want to keep restarting to figure out which one did it, but I think that's the problem. Splunk's not handling use of $SPLUNK_HOME for a path in either inputs.conf or server.conf or both.

View solution in original post

Builder

We had the same problem and this fixed it. Some notes:

use the "$SPLUNKHOME/bin/splunk cmd btool server list --debug" and "$SPLUNKHOME/bin/splunk cmd btool input list --debug" tools to make sure your running config don't have any other places where the relative path still have the environment variable. That was the case with us and we figured that out after making the suggested changes and still getting the sendemail error.

Also if your SH are clustered, make sure you make any necessary changes at the cluster master as well. Ideally, this would be the ONLY place you need to make changes.

0 Karma

Builder

the X509loadcertcrlfile method/function seems like it's inside libcrypto and isn't any exposed python library. We don't have any CRL file.

I remember a long time ago having weird problems with either inputs.conf, server.conf or both when I used relative paths as the documentation suggested. Or maybe it was using $SPLUNK_HOME rather than relative paths. I can't remember. Either way it was unexpected at the time.

Out of curiosity, I was wondering if perhaps Splunk is passing $SPLUNKHOME/ directly into the libcrypto routines and not expanding that variable first. So I changed both the inputs.conf and server.conf SSL entries to not use $SPLUNKHOME but explicitly list the path out. Then I restarted. Then I stopped having the problem.

It's late and I don't want to keep restarting to figure out which one did it, but I think that's the problem. Splunk's not handling use of $SPLUNK_HOME for a path in either inputs.conf or server.conf or both.

View solution in original post

Engager

Make sure you check for relative paths in all apps!

I also had this problem with sendemail erroring out when I had the following in my server.conf:

[sslConfig]
sslRootCAPath = $SPLUNK_HOME/blah

Changing this to:
sslRootCAPath = /opt/splunk/blah

fixed my issues except on one server where sslRootCAPath was defined in multiple apps. To fix the sendemail problem, I had to change the path in all apps where it was defined with relative paths, even if the apps didn't take precedence or show up in btool.

Motivator

I had the sam issue on 6.62 we upgraded to 7.0.1 now and it is fixed.

0 Karma

Explorer

I'm adding a comment here for anyone else encountering this issue, which is apparently going to be fixed in 6.5.3. Note this is assuming you do not use ssl or tls when sending email.

I applied the fixes to my server.conf but the error above continued to occur, even despite working with support to try to identify the problem. After getting all the way to the development level, we discovered that we needed to force Splunk to run a successful search with email without ssl or tls enabled before any other emails could succeed. You can do this from the GUI by making the changes, restarting the Splunk service, and running a search like the following:

head 100 | top 5 host | sendemail to="youremailaddress@whatever.com" use_ssl=0 use_tls=0

If you get email, re-run this search without the usessl or usetls parameters and verify you still get it.

The developer theorized that due to both the bug of not expanding the variable and how the sendemail.py script works in 6.5.x (it still initializes SSL context even if you don't use SSL, in preparation for if your use_ssl is set to enabled), that if it fails, it will have a remnant of that failure in opening the cert file and subsequently fail every other attempt to send email. By forcing a successful sending of mail in bypassing ssl and tls, you basically complete the mail process and clear anything that was cached or stored from before, and email alerts succeed once again.

So if you try this solution and it still doesn't work, try it with the search I've indicated and see if that clears the cert open failure.

Contributor

Thank you! This also fixed our issue when generating new SHA256 self-signed certs.

0 Karma

SplunkTrust
SplunkTrust

$SPLUNK_HOME should work regardless of what config file you use it in. This tells me the OS doesn't have the environmental variable set correctly. the variable is used everywhere so no doubt you'll be facing other problems too.

As for executing the Python command while loading splunks Python environment, I believe the correct method is $SPLUNKHOME/bin/splunk python /path/to/script not $SPLUNKHOME/bin/python /path/to/script

0 Karma

Contributor

mfrost8, please change your comment to an Answer so it can be accepted and upvoted!

Explorer

Thanks a lot for this comment. This is exactly my case as I had relative paths in my server.conf file. Once changed to absolute paths it worked!
Thanks again.

0 Karma

Explorer

I have the very same error on sendemail command. I suspect the cause of that is that we recently secured Splunk with our own certificates and this somehow broke things.
I don't believe that missing packages (like com.Intersplunk) are the true reason. When you execute:
python sendemail.py
this uses your python installation on the machine (I checked and got the same results).
But if I execute:
$SPLUNK_HOME/bin/python sendemail.py
I get the other number of errors on a package import.
In my case the python.log contains the row numbers in the sendemail.py script when failing as well, which lead me to:

# send the mail
if not usessl:
smtp = secure
smtplib.SecureSMTP(host=server)
else:
smtp = securesmtplib.SecureSMTPSSL(host=server, sslContext=ctx)

So it breaks here for me.

0 Karma

Builder

Thanks, takemusu.

We've been using internally signed certificates for quite some time, although if memory serves we recently updated them from SHA1-signed to SHA256-signed but I think that was several months ago.

I've opened a case on this one and will point Splunk support to your comment for further info.

0 Karma

Communicator

com.Intersplunk is misisng and it is a package part of Splunk's python sdk. For some reason it is not loading properly. Have you upgraded Splunk in recent times or a fresh install?

Also check if there is a package with that name here SPLUNK_HOME$/lib/python2.7/site-packages/splunk.

0 Karma

Builder

Isn't the python SDK an optional package? That is, on one of these servers I had developed a python modular input and had to install the SDK but I haven't done that on any of the others.

Yes, these's lots of contents in that site-packages/splunk directory. Just to be certain I'm being clear about this, that python exception output was just me trying to see what happens if I ran the system python command against that sendemail.py file. In some cases I've received helpful output there (at least for non-Splunk python commands). In this case, I'm not surprised it failed as I'm not running the Splunk-python with it. Also note that I tried doing that same thing (python sendemail.py) on a completely different sandbox machine (also Linux, but a different architecture and a pretty recent clean install) and got the same results.

Yes, these servers have been updated pretty regularly.

thanks

0 Karma