Developing for Splunk Enterprise
Highlighted

Search returns only 50000 events in Python script?

New Member

Hi,

We are using the below python script to get the results from Splunk but the problem is that through UI we are getting more than 6lakh records. However, through API we are getting only 50000 records.

Please help - what do I need to add in below script to get all the records?

import urllib
import httplib2
import time
import re
from time import localtime,strftime
from xml.dom import minidom
import json
baseurl = 'https://localhost:8089'
username = ''
password = ''
myhttp = httplib2.Http()

#Step 1: Get a session key
servercontent = myhttp.request(baseurl + '/services/auth/login', 'POST',
                            headers={}, body=urllib.urlencode({'username':username, 'password':password}))[1]
sessionkey = minidom.parseString(servercontent).getElementsByTagName('sessionKey')[0].childNodes[0].nodeValue
print "====>sessionkey:  %s  <====" % sessionkey 

#Step 2: Create a search job    
searchquery = 'index="_internal" | head 10'
if not searchquery.startswith('search'):
searchquery = 'search ' + searchquery

searchjob = myhttp.request(baseurl + '/services/search/jobs','POST',
headers={'Authorization': 'Splunk %s' % sessionkey},body=urllib.urlencode({'search': searchquery}))[1]
sid = minidom.parseString(searchjob).getElementsByTagName('sid')[0].childNodes[0].nodeValue
print "====>sid:  %s  <====" % sid

#Step 3: Get the search status    
myhttp.add_credentials(username, password)
servicessearchstatusstr = '/services/search/jobs/%s/' % sid
isnotdone = True
while isnotdone:
    searchstatus = myhttp.request(baseurl + servicessearchstatusstr, 'GET')[1]
    isdonestatus = re.compile('isDone">(0|1)')
    isdonestatus = isdonestatus.search(searchstatus).groups()[0]
    if (isdonestatus == '1'):
        isnotdone = False
print "====>search status:  %s  <====" % isdonestatus

#Step 4: Get the search results
services_search_results_str = '/services/search/jobs/%s/results?output_mode=json&count=0' % sid
searchresults = myhttp.request(baseurl + services_search_results_str, 'GET')[1]
print "====>search result:  [%s]  <====" % searchresults

 

Labels (2)
0 Karma
Highlighted

Re: search returns only 50000 events

Motivator
0 Karma
Highlighted

Re: search returns only 50000 events

New Member

@kmorris_splunk Yes,I tried but its not working

0 Karma
Highlighted

Re: search returns only 50000 events

Communicator

Since applying that change, have you restarted the Splunk instance?

0 Karma
Highlighted

Re: search returns only 50000 events

SplunkTrust
SplunkTrust

Hi,

For large dataset export, please use jobs/export endpoint https://docs.splunk.com/Documentation/Splunk/7.2.6/RESTREF/RESTsearch#search.2Fjobs.2Fexport

0 Karma
Highlighted

Re: search returns only 50000 events

New Member

Hi @harsmarvania57

Can you please help me how to implement it in the above code?

I am new to this one any help would be much appreciated.

0 Karma
Highlighted

Re: search returns only 50000 events

SplunkTrust
SplunkTrust
0 Karma
Highlighted

Re: search returns only 50000 events

New Member

@harsmarvania57 Thanks for sharing the link.I was thinking to add the loop in my mentioned code to take count as 50000 and offset as 0 then count as 50000 and offset as 50000 and so on....I am not sure how to add this loop in my code.Can you please help me with that?

0 Karma
Highlighted

Re: search returns only 50000 events

SplunkTrust
SplunkTrust

You'll not able to achieve this using loop because results endpoint return only 50000 events. If you want to achieve this using export endpoint with Splunk Python SDK then let me know and I'll provide script.

0 Karma
Highlighted

Re: search returns only 50000 events

New Member

Hi @harsmarvania57 sure please share the script that would be great

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.