Splunk Dev

How do I find scheduled saved searches where external python commands failed?

New Member

I have a scheduled saved search which runs every 15mins. This search calls a custom python external command that does some HTTP calls.

index=main | head 100 | mypythoncommand | collect index=mysummaryindex

Sometimes the the external command fails (say because the service it's calling is down and I get an exception doing the request), in which case I log the error and exit with a failure code.

Imagine the python external command looks something like this:

   mypythoncommand.py does an http call which sometimes fails
import sys
import splunk.Intersplunk
import requests

def main():
    (isgetinfo, sys.argv) = splunk.Intersplunk.isGetInfo(sys.argv)
    if isgetinfo:
        splunk.Intersplunk.outputInfo(False, False, True, False, None, False)

    results, dummyresults, settings = splunk.Intersplunk.getOrganizedResults()

        response = requests.get("https://www.example.com",
                                timeout=30, verify=false)


if __name__ == "__main__":

When you run this search from Splunk Web, the search fails with a descriptive error, and the search.log has the details. I see an error like:

command="mypythoncommand", (<type 'exceptions.Exception'>, Exception(u'Non-OK response from external mypythoncommand service: {"message":"Something bad"}',), <traceback object at 0x7faa83763200>)

However, I can't find any log entries in _internal or _audit that show that the search failed. In fact, _internal shows that the search completed with status success ( source = /opt/splunk/var/log/splunk/scheduler.log sourcetype=scheduler )

So far I've tried the _audit and _internal indexes, but no luck. Is there any other way to determine which scheduled saved search runs failed with an error?

0 Karma

Revered Legend

Give this a try

index=_internal  sourcetype=splunk_python OR sourcetype=scheduler | rex "(Saved Search \[|savedsearch_name=\")(?<SavedSearchName>[^\"\]]+)" | transaction SavedSearchName startswith=sourcetype=splunk_python endswith=sourcetype=scheduler
0 Karma
Get Updates on the Splunk Community!

Get ready to show some Splunk Certification swagger at .conf24!

Dive into the deep end of data by earning a Splunk Certification at .conf24. We're enticing you again this ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Now On-Demand Join us to learn more about how you can leverage Service Level Objectives (SLOs) and the new ...

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...