All Apps and Add-ons

sendresults debug logging

ashleyherbert
Communicator

Hi,
We use the sendresults command for some alerting, and we are having an issue where occasionally the sendresults.py script is hanging, which causes the scheduled search to never finish, and then the subsequent scheduled searches are not run. This happens a couple of times per week, and is causing our alerting to be very unreliable.

I suspect it's a problem with our SMTP server not responding, but I want to turn on some debug logging inside the sendresults.py to figure out where this is hanging. I've figured out I can change the "level=logger.INFO" to DEBUG and that prints out extra debug statements for the Splunk stuff, but it doesn't print out anything about the SMTP connection.

The library that is being used (smtplib) has a function "SMTP.set_debuglevel(True)" which should do this. Do you know if it is possible to add this setting and have it log to the sendresults.log?

Appreciate the help in advance.

Thanks,
Ash

Tags (1)
0 Karma
1 Solution

mockd
Path Finder

Hi,

Yes, the quickest hack you can make to the code is to enable the smtplib debug flag. If you add it, then the output of the smtp session will go to stderr which is picked up in the search log for the job. You can view it in the job inspector/search log. It will be mixed in with all of the other search job output, but you should be at least be able to see what is going on.

You can add it in the sendemail function in sendresults.py as follows:

    try:
            # send the mail
            if not use_ssl:
                smtp = smtplib.SMTP(server)
            else:
                smtp = smtplib.SMTP_SSL(server)

            smtp.set_debuglevel(True)

            if use_tls:
               smtp.ehlo()
               smtp.starttls()
            if len(username) > 0 and len(password) >0:
               smtp.login(username, password)

It's also a great idea for a new feature for the command to capture the SMTP logs if possible.

View solution in original post

mockd
Path Finder

Hi,

Yes, the quickest hack you can make to the code is to enable the smtplib debug flag. If you add it, then the output of the smtp session will go to stderr which is picked up in the search log for the job. You can view it in the job inspector/search log. It will be mixed in with all of the other search job output, but you should be at least be able to see what is going on.

You can add it in the sendemail function in sendresults.py as follows:

    try:
            # send the mail
            if not use_ssl:
                smtp = smtplib.SMTP(server)
            else:
                smtp = smtplib.SMTP_SSL(server)

            smtp.set_debuglevel(True)

            if use_tls:
               smtp.ehlo()
               smtp.starttls()
            if len(username) > 0 and len(password) >0:
               smtp.login(username, password)

It's also a great idea for a new feature for the command to capture the SMTP logs if possible.

ashleyherbert
Communicator

Thanks for the reply mockd, I was on the right path but didn't know where I'd see that output. I've added the line in and I can see the debug output in the search log in the job inspector, so it looks good. Now I just have to wait for it to happen again!

Thanks heaps for your help!

0 Karma
Get Updates on the Splunk Community!

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...

Reminder! Splunk Love Promo: $25 Visa Gift Card for Your Honest SOAR Review With ...

We recently launched our first Splunk Love Special, and it's gone phenomenally well, so we're doing it again, ...