Splunk Dev

Only 100 Results return with python API query

wibbs
Explorer

Hello there,
I'm still newer to Splunk (and python which doesn't help). I used the code from the search and poll results code on the sdk page. I can't seem to figure out how to get more than 100 results. Here's the code

#set login settings above    
kwargs_normalsearch = {"exec_mode": "normal", "max_count" : "20000"}

searchquery_normal = "search *"
job = service.jobs.create(searchquery_normal,  **kwargs_normalsearch)

# A normal search returns the job's SID right away, so we need to poll for completion
while True:
    job.refresh()
    stats = {"isDone": job["isDone"],
             #"doneProgress": float(job["doneProgress"])*100,
              #"scanCount": int(job["scanCount"]),
              "eventCount": int(job["eventCount"]),
              "resultCount": int(job["resultCount"])}
    #status = ("\r%(doneProgress)03.1f%%   %(scanCount)d scanned   "
    status =("\r%(eventCount)d matched   %(resultCount)d results") % stats

    sys.stdout.write(status)
    sys.stdout.flush()
    if stats["isDone"] == "1":
        sys.stdout.write("\n\nDone!\n\n")
        break
    sleep(2)

# Get properties of the job
print "Search job properties"
print "Search job ID:        ", job["sid"]
print "The number of events: ", job["eventCount"]
print "The number of results:", job["resultCount"]
print "Search duration:      ", job["runDuration"], "seconds"
print "This job expires in:  ", job["ttl"], "seconds"
print "------------------------------------------\n"
print "Search results:\n"

num_results=0
# Get the results and display them
newFile = open("splunkResults.txt", 'w')

x = 0
for result in results.ResultsReader(job.results()):
    x += 1
    print x

x always ends up being 100. (see bottom of code). I'm assuming i'm just over looking something but i can't figure out for the life of me what it is.

forgot to give you the output of the script

OUTPUT:

5000 matched   5000 results

Done!

Search job properties
Search job ID:         1376943997.140464
The number of events:  5000
The number of results: 5000
Search duration:       127.888000 seconds
This job expires in:   150000 seconds

Search results:

100
Tags (2)

rafamss
Contributor

Hi wibbs,

I did this way and worked fine to me.

kwargs_options = {"count" : 0}
response = service.job(sid).results(**kwargs_options)

ifeldshteyn
Communicator

Solved: job.results(count=0) fixed it for me.

In my case splunk seems to completely ignore the count variable. I have it set to 1 or 10000, it will always return 100.

From the log...

POST request to https://XXXXXXX:8089/services/search/jobs/ (body: {'count': 10000, 'search_mode': 'normal', 'search': 'search index = myindex earliest=-30m'})

Still returns 100. The job["resultCount"] value says there are 549 results.

If I go to the search job id site

https://XXXXXXX:8089/services/search/jobs/1466455902.9551_2288E5C9-03DA-4BDF-AE92-735977C5CE06/resul... the result offset goes from 0 to 99. So it is just reading from that unfortunately.

If this is due to limits.conf - what exact variable must be updated?

cegoes
Explorer

This also fixed the issue I was having. Thank you!

0 Karma

himanshusinha1
Explorer

Hi All,

Try Below:

kwargs_normalsearch = {"exec_mode": "normal", "count" : 10000}

for result in results.ResultsReader(job.results(**kwargs_normalsearch)):
x += 1
print x

wibbs
Explorer

Should have read a bit closer. I thought this was a kwargs issue, but turns out it's probably a splunk config issue. Anyway, i used this - http://dev.splunk.com/view/SP-CAAAEE5#paginating to get more than 100 results.

wibbs
Explorer

46 views and no idea's?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...