topic Re: Same query run multiple times returns different results in Splunk Search

Same query run multiple times returns different results

rey123 — Sun, 15 Sep 2019 01:50:41 GMT

I got a different result count when I executed this query a week before, and when I executed it today. The first time, the query returned 16 records, today, it returned 21! How is this possible? I ran the search for the same absolute time period both times. If it helps, I experienced similar inconsistent results with another query on the same search head. There are no errors in the search results that could point to any suppressed events:

servername=abc* sourcetype=bq
| rex "java\.\S+\.(?P<Var1>[ A-Z]+(Err))"
| rex field=_raw "(?<Var2>com\.jss\S*\.\S+)\.[A-Z]\S+\((?<Var3>\w+)\.java:(?<Var4>\d+)\)"
| search Var1=NNN
| eval Var3=coalesce(Var3, "No Var3"), Var4=coalesce(Var4, "No Var4"), Var3=Var3. "." .Var4
| search Var3=*
| stats count by Var1, Var3, Var2

I have already spent many hours trying to troubleshoot this, so any pointers would be very helpful. Thank you!

Re: Same query run multiple times returns different results

rey123 — Sun, 15 Sep 2019 03:04:52 GMT

Tagging @somesoni2, @woodcock as they have been very helpful with such questions before and this is a little urgent. Thank you

P.S. - This is on Splunk Enterprise v7.1.6

Re: Same query run multiple times returns different results

woodcock — Sun, 15 Sep 2019 15:31:26 GMT

This is 1 of 2 problems:
1: The events are arriving late. Sometimes the box is completely off, or offline, or Splunk is not running and then it comes back and the events come flooding in late.
2: If you are using an accelerated datamodel, this usually runs behind about 3 minutes but sometimes WAY more than that, especially if you rest it.
You can compare _indextime against _time to differentiate between the 2.

Re: Same query run multiple times returns different results

rey123 — Sun, 15 Sep 2019 17:38:14 GMT

Thank you for your reply, @woodcock, would you mind sharing how I could compare _indextime against _time? Do I remove the stats statement at the end of my query and simply append _indextime and _time to the search statement? Thanks!

Re: Same query run multiple times returns different results

woodcock — Sun, 15 Sep 2019 19:20:01 GMT

Download the Meta Woot! app and it will make it easy to see.

Re: Same query run multiple times returns different results

rey123 — Sun, 15 Sep 2019 19:31:29 GMT

Thanks, can this be installed by someone who's not a Splunk Admin? From a quick online check, it seemed not, but maybe I'm mistaken.