Different events number while searching via python sdk

osnathy83 — Wed, 21 Jul 2021 13:35:47 GMT

Hi,

I am using python SDK to search with this configuration:

query_kwargs = {'earliest_time': earliest,
                'latest_time': latest,
                'results_preview': False,
                'search_mode': 'normal',
                'status_buckets': 2
                }
job =splunk_client.jobs.create(query, **query_kwargs)

As the Splunk documentation (https://dev.splunk.com/enterprise/docs/devtools/python/sdk-python/howtousesplunkpython/howtorunsearchespython/)

I do the follow:

while True:
while not job.is_ready():
pass
 stats = {
'isDone': job['isDone'],
 'doneProgress': job['doneProgress'],
 'scanCount': job['scanCount'],
 'eventCount': job['eventCount'],
 'resultCount': job['resultCount']
}

progress = float(stats['doneProgress'])*100
 scanned = int(stats['scanCount'])
matched = int(stats['eventCount'])
result_count = int(stats['resultCount'])

if verbose:
status = ("\r%03.1f%% | %d scanned | %d matched | %d results" % (progress, scanned, matched, result_count))
sys.stdout.write(status)
sys.stdout.flush()

if job["isDone"] == "1":
if verbose:
sys.stdout.write("\n")
break
 time.sleep(2)

Then once the job is finished I do this:

offset = 0
max_event_count = 50000

total_results = []
first_50k_results = self.get_results(job, offset, max_event_count)
total_results.extend(first_50k_results)

while offset <= number_of_results:
    offset += max_event_count
    intermediate_result = self.get_results(job, offset, max_event_count)
    total_results.extend(intermediate_result)

def get_results(self, job, offset, max_event_count):
    logger.info("collecting results,please wait . . ")
    results_list = []
    kwargs_paginate = {"count": max_event_count, "offset": offset}
    for result in results.ResultsReader(job.results(**kwargs_paginate)):
        results_list.append(result)
    return results_list

The issue is that the number of events that the python search return is different from the number of events that the search in the Splunk console return.

Can you please advise what I am doing wrong?

Please note that I am using explicit index= in my search

topic Different events number while searching via python sdk in Splunk Search

Different events number while searching via python sdk