sendresults debug logging

[ashleyherbert]

Hi,
We use the sendresults command for some alerting, and we are having an issue where occasionally the sendresults.py script is hanging, which causes the scheduled search to never finish, and then the subsequent scheduled searches are not run. This happens a couple of times per week, and is causing our alerting to be very unreliable.

I suspect it's a problem with our SMTP server not responding, but I want to turn on some debug logging inside the sendresults.py to figure out where this is hanging. I've figured out I can change the "level=logger.INFO" to DEBUG and that prints out extra debug statements for the Splunk stuff, but it doesn't print out anything about the SMTP connection.

The library that is being used (smtplib) has a function "SMTP.set_debuglevel(True)" which should do this. Do you know if it is possible to add this setting and have it log to the sendresults.log?

Appreciate the help in advance.

Thanks,
Ash

[mockd · Accepted solution]

Hi,

Yes, the quickest hack you can make to the code is to enable the smtplib debug flag. If you add it, then the output of the smtp session will go to stderr which is picked up in the search log for the job. You can view it in the job inspector/search log. It will be mixed in with all of the other search job output, but you should be at least be able to see what is going on.

You can add it in the sendemail function in sendresults.py as follows:

    try:
            # send the mail
            if not use_ssl:
                smtp = smtplib.SMTP(server)
            else:
                smtp = smtplib.SMTP_SSL(server)

            smtp.set_debuglevel(True)

            if use_tls:
               smtp.ehlo()
               smtp.starttls()
            if len(username) > 0 and len(password) >0:
               smtp.login(username, password)

It's also a great idea for a new feature for the command to capture the SMTP logs if possible.

[ashleyherbert]

Thanks for the reply mockd, I was on the right path but didn't know where I'd see that output. I've added the line in and I can see the debug output in the search log in the job inspector, so it looks good. Now I just have to wait for it to happen again!

Thanks heaps for your help!