topic Re: Python SDK save search to csv in Splunk Dev

Python SDK save search to csv

to914868 — Wed, 04 Jul 2018 10:42:50 GMT

I want to use splunklib to run a one-off Splunk query and save it to csv.
I'm testing with a small query (a single visitId) of 8 events only.
The result is returned immediately in Splunk UI but I have problems getting the result from the python-sdk.

My problems with splunklib are:
- service.jobs.export() query does not complete because it keeps repeating the same 8 event results over and over again
- service.jobs.oneshot() query does not finish and returns no result

I tried adding the search parameters "preview"=False, i.e.

kwargs_export = { "search_mode": "normal","preview": True }
rr = results.ResultsReader(service.jobs.export(query,**kwargs_export ))

The only effect is that neither option returns anything anymore, since the queries are not completing.

import splunklib.client as client
import splunklib.results as results
service = client.connect( 
               host=HOST, 
               port=8089,
               username=USERNAME,
               password=PWD )
    query= """search index=xxx application="xxx" sourcetype=xxx| 
    spath visitId  | join type ..."""
    rr = results.ResultsReader(service.jobs.export(query))

    for item in rr:
        for key in item.keys():
            print(key, len(item[key]), item[key])

I tried the same with oneshot

kwargs_oneshot = {'output_mode': 'csv',"search_mode": "normal"}
oneshotsearch_results = service.jobs.oneshot(query, **kwargs_oneshot)
 f=open('myresults.csv', 'w')
 f.write(oneshotsearch_results.read())

This creates a csv file but has no content at all. I think .read is deprecated.
Any suggestions ?
All I want is to save the query results to .csv ONCE using the library.
Thanks!

Re: Python SDK save search to csv

evuk — Mon, 16 Jul 2018 19:41:13 GMT

try:

kwargs_export = { "output_mode": "csv"}
rr = service.jobs.export(query)

for item in rr:
    print(item)

I think that you shouldn't need to convert the result into resultsreader because it already is one.

Re: Python SDK save search to csv

poete — Tue, 17 Jul 2018 06:37:10 GMT

Hello @to914868,

please add f.close() on the next line after f.write(oneshotsearch_results.read())

I think the content is not flushed to the file.

@to914868, please accet this answer in order for other users to find more easily the answer to this question.

Re: Python SDK save search to csv

to914868 — Fri, 10 Aug 2018 18:28:24 GMT

Thanks @poete!

Here is what I used in the end

results_kwargs = {
 "earliest_time": "-40min",
 "latest_time": "now",
 "search_mode": "normal",
 "output_mode": "csv"
}
oneshotsearch_results = service.jobs.oneshot(query, **results_kwargs)
f=open('myresults.csv', 'w')
f.write(oneshotsearch_results.read())
f.close()

Re: Python SDK save search to csv

kkrishnan_splun — Tue, 30 Oct 2018 18:31:58 GMT

This works.

Re: Python SDK save search to csv

pchp348 — Mon, 10 Dec 2018 14:18:13 GMT

This is working fine , But i could not fetch all the results in csv. Kindly provide me the solution for this question
https://answers.splunk.com/answers/708529/export-to-csv-is-not-fetching-all-the-results-pyth.html?minQuestionBodyLength=80