Why are REST API searches returning intermittent results?

[ltrand]

I'm having an weird situation where REST queries sometimes pull results and sometimes don't. We've even tried limiting the search to just return 10 results with | head 10 after the gather stage. I see the traffic in PCAP for all request attempts, but some requests just fail to return any results. For those that fail, I don't see any log entries for the search. For all that returned results, I see log entries for the search activity. The same search can fail and then succeed or vice versa.

What should I start investigating next? Has anyone seen an issue like this before?

[javiergn]

Have you tried adding the following to your query?

splunk_server=local

For instance, who am I?

| rest /services/authentication/current-context splunk_server=local | table username

I remember getting weird results when running REST queries on the DMC server and forcing the queries to run against the local server fixed the problem

[jplumsdaine22]

What does the _internal index say about those SIDS ?

[ltrand]

That's what's killer, the ones that fail have no entries for the user in _internal at all. It's like it didn't even register the request as valid.

[jplumsdaine22]

Whats in the HTTP response? EG: Do you get any response at all from Splunk or are your requests just timing out.