topic REST API Python - Issue with pulling results before search job is done. in Getting Data In

REST API Python - Issue with pulling results before search job is done.

sd248011 — Mon, 28 Sep 2020 13:34:05 GMT

I wrote a script in Python to run a search query and return the results. The code to send the search query is:

sid1 = httplib2.Http(disable_ssl_certificate_validation=True).request(baseurl + '/services/search/jobs','POST',
headers={'Authorization': 'Splunk %s' % sessionKey},body=urllib.urlencode({'search': searchQuery1}))[1]

The code to return the results is:

response1 = httplib2.Http(disable_ssl_certificate_validation=True).request(baseurl + '/services/search/jobs/' + slicesid1 + '/results?count=0','GET',
headers={'Authorization': 'Splunk %s' % sessionKey},body=urllib.urlencode({'search': searchQuery1}))[1]

The issue is that if the results request is made before the result query is completed, no results will be returned. I have fiddled with sleep time which allows me to return results but that isn't very efficient as the sleep time can be too long or too short.

I know there is a field called dispatchState that is RUNNING during the search which changes to DONE when it is completed. How can I put some code in after the initial search query that will continuously check to see if the job is RUNNING or DONE and then once DONE, the results query will run?

Re: REST API Python - Issue with pulling results before search job is done.

Neeraj_Luthra — Thu, 21 Mar 2013 15:49:52 GMT

You can write a while loop and continue to refresh the job and check for isDone property on the job. Exit when the property value changes to 1.

Checkout the Python SDK and code sample for this while loop at http://dev.splunk.com/view/SP-CAAAEE5#normaljob.

Re: REST API Python - Issue with pulling results before search job is done.

sd248011 — Thu, 21 Mar 2013 18:56:25 GMT

Thanks much that is exactly what I needed. Only problem I have now is it is not returning all of my results. Can you let me know how I would add in the count=0 argument on the results line that is:

for result2 in results.ResultsReader(job.results()):
print result2

Re: REST API Python - Issue with pulling results before search job is done.

Neeraj_Luthra — Thu, 21 Mar 2013 19:43:25 GMT

Try this:

body=urllib.urlencode({'search': searchQuery1, 'count': 0})

Re: REST API Python - Issue with pulling results before search job is done.

sd248011 — Thu, 21 Mar 2013 20:01:32 GMT

I was able to use the count argument in my original code at the top and pull all of the results. Now I am using the code in the link you gave me with the python SDK and can't figure out how to add in the count=0 argument. In your link:

for result2 in results.ResultsReader(job.results()):
print result2

The above code is to pull the results now.

Re: REST API Python - Issue with pulling results before search job is done.

Neeraj_Luthra — Mon, 28 Sep 2020 13:34:39 GMT

How about this ...

kwargs_results = {"count": 0}

search_results = job.results(**kwargs_results)