I got a different result count when I executed this query a week before, and when I executed it today. The first time, the query returned 16 records, today, it returned 21! How is this possible? I ran the search for the same absolute time period both times. If it helps, I experienced similar inconsistent results with another query on the same search head. There are no errors in the search results that could point to any suppressed events:
servername=abc* sourcetype=bq | rex "java\.\S+\.(?P<Var1>[ A-Z]+(Err))" | rex field=_raw "(?<Var2>com\.jss\S*\.\S+)\.[A-Z]\S+\((?<Var3>\w+)\.java:(?<Var4>\d+)\)" | search Var1=NNN | eval Var3=coalesce(Var3, "No Var3"), Var4=coalesce(Var4, "No Var4"), Var3=Var3. "." .Var4 | search Var3=* | stats count by Var1, Var3, Var2
I have already spent many hours trying to troubleshoot this, so any pointers would be very helpful. Thank you!
This is 1 of 2 problems:
1: The events are arriving late. Sometimes the box is completely off, or offline, or Splunk is not running and then it comes back and the events come flooding in late.
2: If you are using an accelerated datamodel, this usually runs behind about 3 minutes but sometimes WAY more than that, especially if you rest it.
You can compare
_time to differentiate between the 2.
Thank you for your reply, @woodcock, would you mind sharing how I could compare
_time? Do I remove the
stats statement at the end of my query and simply append
_time to the
search statement? Thanks!