topic Re: Why are REST API searches returning intermittent results? in Splunk Search

Why are REST API searches returning intermittent results?

ltrand — Thu, 19 Nov 2015 20:44:33 GMT

I'm having an weird situation where REST queries sometimes pull results and sometimes don't. We've even tried limiting the search to just return 10 results with | head 10 after the gather stage. I see the traffic in PCAP for all request attempts, but some requests just fail to return any results. For those that fail, I don't see any log entries for the search. For all that returned results, I see log entries for the search activity. The same search can fail and then succeed or vice versa.

What should I start investigating next? Has anyone seen an issue like this before?

Re: Why are REST API searches returning intermittent results?

jplumsdaine22 — Tue, 01 Dec 2015 12:50:20 GMT

What does the _internal index say about those SIDS ?

Re: Why are REST API searches returning intermittent results?

ltrand — Tue, 01 Dec 2015 16:40:03 GMT

That's what's killer, the ones that fail have no entries for the user in _internal at all. It's like it didn't even register the request as valid.

Re: Why are REST API searches returning intermittent results?

jplumsdaine22 — Tue, 01 Dec 2015 16:51:05 GMT

Whats in the HTTP response? EG: Do you get any response at all from Splunk or are your requests just timing out.

Re: Why are REST API searches returning intermittent results?

javiergn — Wed, 02 Dec 2015 14:36:21 GMT

Have you tried adding the following to your query?

splunk_server=local

For instance, who am I?

| rest /services/authentication/current-context splunk_server=local | table username

I remember getting weird results when running REST queries on the DMC server and forcing the queries to run against the local server fixed the problem