Inconsistency between search results between Splunk UI and Rest API & REST API itself

[siddharth1479]

Hi Community,
I'm trying to extract search results using REST API and I'm facing the following problem.
1. I'm using the curl command: curl --location --request POST 'https://XXXXXX/services/search/jobs/export' \ --data-urlencode 'search=search index=uam user="abcd" event=auth earliest="01/16/2020:00:00:00" latest=now() | eval access_time=strftime(_time, "%Y-%d-%m %H:%M:%S") | stats count by access_time user status | fields - count' \ --data-urlencode 'output_mode=json'. The results I get are different every time I fire up the API. Sometime there are 10 results, sometimes 20, 15, and so on; i.e. inconsistent.

1. When I use the same searchquery in the Splunk UI, I get the reults, which are different from the results i get from the API call, which is desired. Search result I use is: index=uam user="abcd" event=auth earliest="01/16/2020:00:00:00" latest=now() | eval access_time=strftime(_time, "%Y-%d-%m %H:%M:%S") | stats count by access_time user status | fields - count.

Along with that, when i get back the json results, there is a field "preview" which have values "true" or "false", I assume that the results with preview: true might be shown in the Splunk UI.

Result from the API call:
`{
"preview": true,
"offset": 9,
"result": {
"access_time": "2020-15-01 18:06:21",
"user": "adcgwjv_ahubt_ext1",
"status": "success"
}
}

{
"preview": false,
"offset": 9,
"result": {
"access_time": "2020-15-01 18:06:21",
"user": "adcgwjv_ahubt_ext1",
"status": "success"
}
}
`

Even after using | dedup access_time, i get the repeated results on API call, but works fine in the Splunk UI.

Please help.

Thanks,
Sid

[jkat54 · Accepted solution]

Bear in mind that when you dispatch a search via UI you are having RBAC applied as well as KOs from the App Context you are in.

When you dispatch a search via REST, you might not be executing the search in the correct name space (servicesNS/user/app/) or you may be executing as a different role.

Please see docs about servicesNS endpoints and try using the same app & user context with your curl command.

[siddharth1479]

Can you elaborate more? I'm fairly new to Splunk and learning to operate. Is there any link i can refer? Also in the above given curl command, can you point out that part of it are you talking about?

Thanks for the help @jkat54.

[jkat54]

https://docs.splunk.com/Documentation/Splunk/8.0.1/RESTTUT/RESTsearches

Follow the above and your problem should go away.

You're currently getting results from "services/search/jobs/export" and you should be using something like "services/search/jobs/SID/results" instead

[siddharth1479]

Yes, now I got to know what you're talking about and what caused the duplication. Thanks for the insights.

So now that you told that, we need to look for "SID" to obtain the results, what i did is created a chained API call which takes sid from one POST call "/services/search/jobs" which gives SID and the using this SID, in to GET call "/services/search/{{sid}}/results".

But again, my question is; Is there anyway which can make this single call or simpler logic for this results generation?

Thanks,
Sid

[jkat54]

You can make the search a scheduled report in splunk and then call the latest results of the scheduled report each time instead.

This would have 2 benefits. It would give fast results with less calls to the API, and it can help reduce
the load your code would place on the system. Since you'd always get the results of the last run, you wouldn't be running new searches all the time with your code.

[siddharth1479]

That's a great idea. You will get almost a real=time solution and also make it more proficient.

Thanks for all the help.

[jkat54]

Anytime!!!

[jkat54]

This is what you're currently doing:

search/jobs/export
https://:/services/search/jobs/export

"Stream search results as they become available."

Which is why the results keep changing / "duplicating".

[efavreau]

Anytime I use latest=now() I expect differing results. How can the UI results be different all together and desired? Feels like some information is missing about how the data is different. For the sake of comparing apples to apples, fix your time frames and do not make them relative.

[siddharth1479]

Hi, Thanks for the reply. Using latest=new() or latest="01/17/2020:11:00:00", the results are inconsistent in both the cases.

And for the missing information, I've edited the post with some more insights.