topic Re: Python SDK: search from job does not use all fields in Splunk Dev

Python SDK: search from job does not use all fields

ksander14 — Wed, 10 Jan 2018 12:50:13 GMT

I have some source named "source1" with field named "field1". This field is not present in all events (field is filled only in 3 percent of events). So, I try to get events with this field using Splunk Python SDK using simple construction from documentation.

search_text = "source=source1 field1=* earliest=-1d@d latest=@d"
job = jobs.create(search_text)
while True:
while not job.is_ready():
pass
stats = {"isDone": job["isDone"],
"doneProgress": float(job["doneProgress"])*100,
"scanCount": int(job["scanCount"]),
"eventCount": int(job["eventCount"]),
"resultCount": int(job["resultCount"])}
status = ("\r%(doneProgress)03.1f%% %(scanCount)d scanned "
"%(eventCount)d matched %(resultCount)d results") % stats
sys.stdout.write(status)
sys.stdout.flush()
if stats["isDone"] == "1":
sys.stdout.write("\n\nDone!\n\n")
break
sleep(2)

But this search, run using Splunk Python SDK, return me 0 results. If I run this search in Splunk, I get right results - events with filled "field1". So, as I understand this situation, search in Splunk Python SDK doesn't see "field1" in my source.

How can I run search with Spunk Python SDK that see all fields in my source?

P.S. For example - if I run "search source=source1 earliest=-1d@d latest=@d | fieldsummary" in Splunk, I get information about 84 fields, in Splunk Python SDK - 81 fields

Re: Python SDK: search from job does not use all fields

micahkemp — Wed, 10 Jan 2018 15:34:11 GMT

Change your search string in the script to search source=source1 field1=* earliest=-1d@d latest=@d.

The REST API, which the SDK uses, does not add the implied search command to the front of your search strings like the UI does.

Also, add | fields <field1> <field2>. This should ensure that Splunk grabs the fields you care about, even if running in fast mode (which may be what is happening here).

Re: Python SDK: search from job does not use all fields

ksander14 — Thu, 11 Jan 2018 12:09:25 GMT

Unfortunately, this did not help

Re: Python SDK: search from job does not use all fields

micahkemp — Thu, 11 Jan 2018 15:12:48 GMT

Updated answer to include fields.

Re: Python SDK: search from job does not use all fields

ksander14 — Fri, 12 Jan 2018 09:11:21 GMT

Thanks for the advice, but it did not help 😞

Re: Python SDK: search from job does not use all fields

burtica — Fri, 12 Jan 2018 15:27:24 GMT

Hi, I have the same problem.
I've updated the search to be search index=alfa | fields field1, field2. Also tried adding the rf field in search_kwargs, but still these fields are not returned, so my search returns no results (although there are of course results).

The code looks like this:

search_kwargs = {
            "exec_mode": "normal",
            "earliest_time": "-8d",
            "latest_time": "now",
            "search_mode": "normal",
            "rf": ["field1", "field2"],
}
search = "search index=alfa | fields field1, field2"
job = jobs.create(search, **search_kwargs)
# Then pool for job completion, and list results....

Re: Python SDK: search from job does not use all fields

burtica — Fri, 12 Jan 2018 15:30:33 GMT

Hi, I have the same problem.
I have also added the fields in the rf field of the search_kwargs as below, but still I am not returned any data, although there is data. The fields field1 and field2 are actually not returned.

search_kwargs = {
            "exec_mode": "normal",
            "earliest_time": "-8d",
            "latest_time": "now",
            "search_mode": "normal",
            "rf": ["field1", "field2"],
}
search = "search index=alfa | fields field1, field2"
job = jobs.create(search, **search_kwargs)
# Then pool for job completion, and list results....

Re: Python SDK: search from job does not use all fields

micahkemp — Fri, 12 Jan 2018 16:29:56 GMT

How is field1 defined? Is it a custom extraction you put in place? Is it shared globally?

Is the account you're using to log in to SplunkWeb the same account you're using for the python script?

I put your code into a python script and it seems to work just fine (I removed the timeframe from the search string for mine):

search_text = "search source=source1 field1=*"
job = service.jobs.create(search_text)
while True:
    while not job.is_ready():
            pass
    stats = {"isDone": job["isDone"],
        "doneProgress": float(job["doneProgress"])*100,
        "scanCount": int(job["scanCount"]),
        "eventCount": int(job["eventCount"]),
        "resultCount": int(job["resultCount"])}
    status = ("\r%(doneProgress)03.1f%% %(scanCount)d scanned "
        "%(eventCount)d matched %(resultCount)d results") % stats
    sys.stdout.write(status)
    sys.stdout.flush()
    if stats["isDone"] == "1":
        sys.stdout.write("\n\nDone!\n\n")
        break
    sleep(2)