topic Inconsistency between search results between Splunk UI and Rest API & REST API itself in Getting Data In

Inconsistency between search results between Splunk UI and Rest API & REST API itself

siddharth1479 — Wed, 30 Sep 2020 03:44:55 GMT

Hi Community,
I'm trying to extract search results using REST API and I'm facing the following problem.
1. I'm using the curl command: curl --location --request POST 'https://XXXXXX/services/search/jobs/export' \ --data-urlencode 'search=search index=uam user="abcd" event=auth earliest="01/16/2020:00:00:00" latest=now() | eval access_time=strftime(_time, "%Y-%d-%m %H:%M:%S") | stats count by access_time user status | fields - count' \ --data-urlencode 'output_mode=json'. The results I get are different every time I fire up the API. Sometime there are 10 results, sometimes 20, 15, and so on; i.e. inconsistent.

When I use the same searchquery in the Splunk UI, I get the reults, which are different from the results i get from the API call, which is desired. Search result I use is: index=uam user="abcd" event=auth earliest="01/16/2020:00:00:00" latest=now() | eval access_time=strftime(_time, "%Y-%d-%m %H:%M:%S") | stats count by access_time user status | fields - count.

Along with that, when i get back the json results, there is a field "preview" which have values "true" or "false", I assume that the results with preview: true might be shown in the Splunk UI.

Result from the API call:
`{
"preview": true,
"offset": 9,
"result": {
"access_time": "2020-15-01 18:06:21",
"user": "adcgwjv_ahubt_ext1",
"status": "success"
}
}

{
"preview": false,
"offset": 9,
"result": {
"access_time": "2020-15-01 18:06:21",
"user": "adcgwjv_ahubt_ext1",
"status": "success"
}
}
`

Even after using | dedup access_time, i get the repeated results on API call, but works fine in the Splunk UI.

Please help.

Thanks,
Sid

Re: Inconsistency between search results between Splunk UI and Rest API & REST API itself

efavreau — Fri, 17 Jan 2020 16:27:51 GMT

Anytime I use latest=now() I expect differing results. How can the UI results be different all together and desired? Feels like some information is missing about how the data is different. For the sake of comparing apples to apples, fix your time frames and do not make them relative.

Re: Inconsistency between search results between Splunk UI and Rest API & REST API itself

siddharth1479 — Fri, 17 Jan 2020 16:36:29 GMT

Hi, Thanks for the reply. Using latest=new() or latest="01/17/2020:11:00:00", the results are inconsistent in both the cases.

And for the missing information, I've edited the post with some more insights.

Re: Inconsistency between search results between Splunk UI and Rest API & REST API itself

jkat54 — Fri, 17 Jan 2020 17:54:56 GMT

Bear in mind that when you dispatch a search via UI you are having RBAC applied as well as KOs from the App Context you are in.

When you dispatch a search via REST, you might not be executing the search in the correct name space (servicesNS/user/app/) or you may be executing as a different role.

Please see docs about servicesNS endpoints and try using the same app & user context with your curl command.

Re: Inconsistency between search results between Splunk UI and Rest API & REST API itself

siddharth1479 — Fri, 17 Jan 2020 18:09:07 GMT

Can you elaborate more? I'm fairly new to Splunk and learning to operate. Is there any link i can refer? Also in the above given curl command, can you point out that part of it are you talking about?

Thanks for the help @jkat54.

Re: Inconsistency between search results between Splunk UI and Rest API & REST API itself

jkat54 — Fri, 17 Jan 2020 20:32:45 GMT

https://docs.splunk.com/Documentation/Splunk/8.0.1/RESTTUT/RESTsearches

Follow the above and your problem should go away.

You're currently getting results from "services/search/jobs/export" and you should be using something like "services/search/jobs/SID/results" instead

Re: Inconsistency between search results between Splunk UI and Rest API & REST API itself

jkat54 — Fri, 17 Jan 2020 20:34:47 GMT

This is what you're currently doing:

search/jobs/export
https://:/services/search/jobs/export

"Stream search results as they become available."

Which is why the results keep changing / "duplicating".

Re: Inconsistency between search results between Splunk UI and Rest API & REST API itself

siddharth1479 — Tue, 21 Jan 2020 17:59:06 GMT

Yes, now I got to know what you're talking about and what caused the duplication. Thanks for the insights.

So now that you told that, we need to look for "SID" to obtain the results, what i did is created a chained API call which takes sid from one POST call "/services/search/jobs" which gives SID and the using this SID, in to GET call "/services/search/{{sid}}/results".

But again, my question is; Is there anyway which can make this single call or simpler logic for this results generation?

Thanks,
Sid

Re: Inconsistency between search results between Splunk UI and Rest API & REST API itself

jkat54 — Tue, 21 Jan 2020 18:07:04 GMT

You can make the search a scheduled report in splunk and then call the latest results of the scheduled report each time instead.

This would have 2 benefits. It would give fast results with less calls to the API, and it can help reduce
the load your code would place on the system. Since you'd always get the results of the last run, you wouldn't be running new searches all the time with your code.

Re: Inconsistency between search results between Splunk UI and Rest API & REST API itself

siddharth1479 — Tue, 21 Jan 2020 21:21:49 GMT

That's a great idea. You will get almost a real=time solution and also make it more proficient.

Thanks for all the help.

Re: Inconsistency between search results between Splunk UI and Rest API & REST API itself

jkat54 — Wed, 22 Jan 2020 12:01:33 GMT

Anytime!!!

Re: Inconsistency between search results between Splunk UI and Rest API & REST API itself

splunkreal — Fri, 18 Apr 2025 16:43:01 GMT

@jkat54 Hello, I find out I get lastTime tstats metadata from export api endpoint ran from CLI and not getting this lastTime field on web search with same query, even if lastTime info is from last year from offline UF. I guess there is maybe web filtering. This only applies to single result though.

Results can be different I guess due to different user/role, app context, api endpoint which may be my case.

Thanks.