topic Re: How to convert Intersplunk's readResults() to dataframe? in Splunk Dev

How to convert Intersplunk's readResults() to dataframe?

agnesramos — Mon, 08 Aug 2016 14:32:46 GMT

Hi,

I am trying to add a Python script as a Splunk custom command and I'm having trouble reading the data from Intersplunk and formatting it as a pandas dataframe. I have:

results = splunk.Intersplunk.readResults()
df = pandas.DataFrame(results)
ip_list = df['ip'].tolist()

So I'm converting the list of dictionaries returned by readResults() to a pandas DF and then extracting what would be the csv "ip" column as a list. But I am getting an error on that last code line.

I have also tried df = pandas.DataFrame.from_records(results) and ip_list = df['ip'].values.tolist(), but it's not working.

I'd appreciate any help.

Re: How to convert Intersplunk's readResults() to dataframe?

danbar6 — Fri, 01 Dec 2017 21:31:23 GMT

Some more info will help - What error are you getting? What does your 'ip' field look like?

umm and also, why are you converting it to a DF just to go back to a list right away?

Re: How to convert Intersplunk's readResults() to dataframe?

selcukozer — Tue, 29 Sep 2020 19:24:26 GMT

You better to use service jobs as following:

Function to Perform a Splunk search

def execute_query(searchquery_normal,
kwargs_normalsearch={"exec_mode": "normal"},
kwargs_options={"output_mode": "csv", "count": 1000000}):
# Execute Search
job = service.jobs.create(searchquery_normal, **kwargs_normalsearch)

# A normal search returns the job's SID right away, so we need to poll for completion
while True:
    while not job.is_ready():
        pass
    stats = {"isDone": job["isDone"], "doneProgress": float(job["doneProgress"])*100, 
             "scanCount": int(job["scanCount"]), "eventCount": int(job["eventCount"]), 
             "resultCount": int(job["resultCount"])}
    status = ("\r%(doneProgress)03.1f%%   %(scanCount)d scanned   " 
              "%(eventCount)d matched   %(resultCount)d results") % stats

    sys.stdout.write(status + '\n')
    sys.stdout.flush()
    if stats["isDone"] == "1":
        sys.stdout.write("\nDone!")
        break
    time.sleep(0.5)

# Get the results and display them
csv_results = job.results(**kwargs_options).read()
job.cancel()

for row in csv_results:
            if row[0] not in (None, ""):
                df = pd.read_csv(StringIO.StringIO(csv_results), encoding='utf8', sep=',', low_memory=False)
                df.to_csv(filename_new, sep=',', encoding='utf-8')
                break
            break

you can find whole project from following:
https://github.com/selcukozer/splunk_python