topic Re: Only 100 Results return with python API query in Splunk Dev

Only 100 Results return with python API query

wibbs — Mon, 19 Aug 2013 20:34:42 GMT

Hello there,
I'm still newer to Splunk (and python which doesn't help). I used the code from the search and poll results code on the sdk page. I can't seem to figure out how to get more than 100 results. Here's the code

#set login settings above    
kwargs_normalsearch = {"exec_mode": "normal", "max_count" : "20000"}

searchquery_normal = "search *"
job = service.jobs.create(searchquery_normal,  **kwargs_normalsearch)

# A normal search returns the job's SID right away, so we need to poll for completion
while True:
    job.refresh()
    stats = {"isDone": job["isDone"],
             #"doneProgress": float(job["doneProgress"])*100,
              #"scanCount": int(job["scanCount"]),
              "eventCount": int(job["eventCount"]),
              "resultCount": int(job["resultCount"])}
    #status = ("\r%(doneProgress)03.1f%%   %(scanCount)d scanned   "
    status =("\r%(eventCount)d matched   %(resultCount)d results") % stats

    sys.stdout.write(status)
    sys.stdout.flush()
    if stats["isDone"] == "1":
        sys.stdout.write("\n\nDone!\n\n")
        break
    sleep(2)

# Get properties of the job
print "Search job properties"
print "Search job ID:        ", job["sid"]
print "The number of events: ", job["eventCount"]
print "The number of results:", job["resultCount"]
print "Search duration:      ", job["runDuration"], "seconds"
print "This job expires in:  ", job["ttl"], "seconds"
print "------------------------------------------\n"
print "Search results:\n"

num_results=0
# Get the results and display them
newFile = open("splunkResults.txt", 'w')

x = 0
for result in results.ResultsReader(job.results()):
    x += 1
    print x

x always ends up being 100. (see bottom of code). I'm assuming i'm just over looking something but i can't figure out for the life of me what it is.

forgot to give you the output of the script

OUTPUT:

5000 matched   5000 results

Done!

Search job properties
Search job ID:         1376943997.140464
The number of events:  5000
The number of results: 5000
Search duration:       127.888000 seconds
This job expires in:   150000 seconds

Search results:

100

Re: Only 100 Results return with python API query

wibbs — Tue, 20 Aug 2013 16:37:58 GMT

46 views and no idea's?

Re: Only 100 Results return with python API query

wibbs — Tue, 20 Aug 2013 20:57:46 GMT

Should have read a bit closer. I thought this was a kwargs issue, but turns out it's probably a splunk config issue. Anyway, i used this - http://dev.splunk.com/view/SP-CAAAEE5#paginating to get more than 100 results.

Re: Only 100 Results return with python API query

himanshusinha1 — Mon, 28 Sep 2020 14:41:16 GMT

Hi All,

Try Below:

kwargs_normalsearch = {"exec_mode": "normal", "count" : 10000}

for result in results.ResultsReader(job.results(**kwargs_normalsearch)):
x += 1
print x

Re: Only 100 Results return with python API query

ifeldshteyn — Mon, 20 Jun 2016 21:03:37 GMT

Solved: job.results(count=0) fixed it for me.

In my case splunk seems to completely ignore the count variable. I have it set to 1 or 10000, it will always return 100.

From the log...

POST request to https://XXXXXXX:8089/services/search/jobs/ (body: {'count': 10000, 'search_mode': 'normal', 'search': 'search index = myindex earliest=-30m'})

Still returns 100. The job["resultCount"] value says there are 549 results.

If I go to the search job id site

https://XXXXXXX:8089/services/search/jobs/1466455902.9551_2288E5C9-03DA-4BDF-AE92-735977C5CE06/results the result offset goes from 0 to 99. So it is just reading from that unfortunately.

If this is due to limits.conf - what exact variable must be updated?

Re: Only 100 Results return with python API query

rafamss — Fri, 01 Jul 2016 15:04:20 GMT

Hi wibbs,

I did this way and worked fine to me.

kwargs_options = {"count" : 0}
response = service.job(sid).results(**kwargs_options)

Re: Only 100 Results return with python API query

cegoes — Sun, 16 Apr 2017 20:44:01 GMT

This also fixed the issue I was having. Thank you!