Splunk Search

Same query run multiple times returns different results

Explorer

I got a different result count when I executed this query a week before, and when I executed it today. The first time, the query returned 16 records, today, it returned 21! How is this possible? I ran the search for the same absolute time period both times. If it helps, I experienced similar inconsistent results with another query on the same search head. There are no errors in the search results that could point to any suppressed events:

servername=abc* sourcetype=bq
| rex "java\.\S+\.(?P<Var1>[ A-Z]+(Err))"
| rex field=_raw "(?<Var2>com\.jss\S*\.\S+)\.[A-Z]\S+\((?<Var3>\w+)\.java:(?<Var4>\d+)\)"
| search Var1=NNN
| eval Var3=coalesce(Var3, "No Var3"), Var4=coalesce(Var4, "No Var4"), Var3=Var3. "." .Var4
| search Var3=*
| stats count by Var1, Var3, Var2

I have already spent many hours trying to troubleshoot this, so any pointers would be very helpful. Thank you!

0 Karma

Esteemed Legend

This is 1 of 2 problems:
1: The events are arriving late. Sometimes the box is completely off, or offline, or Splunk is not running and then it comes back and the events come flooding in late.
2: If you are using an accelerated datamodel, this usually runs behind about 3 minutes but sometimes WAY more than that, especially if you rest it.
You can compare _indextime against _time to differentiate between the 2.

0 Karma

Explorer

Thank you for your reply, @woodcock, would you mind sharing how I could compare _indextime against _time? Do I remove the stats statement at the end of my query and simply append _indextime and _time to the search statement? Thanks!

0 Karma

Esteemed Legend

Download the Meta Woot! app and it will make it easy to see.

0 Karma

Explorer

Thanks, can this be installed by someone who's not a Splunk Admin? From a quick online check, it seemed not, but maybe I'm mistaken.

0 Karma

Explorer

Tagging @somesoni2, @woodcock as they have been very helpful with such questions before and this is a little urgent. Thank you

P.S. - This is on Splunk Enterprise v7.1.6

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!