Inner/outer search results don't match

[ShagVT]

I have a query trying to compare two different time periods, which I do with an inner search ( | append [search <identical query>] ).  But I've run into a curious problem.  If I search over the same period of time in both the inner and outer searches, I get different results.

So if I set the time picker (ie, driving the outer query) to "last week" and my inner query to earliest=-w@w latest=@w, it seems like they should produce identical results since they are identical queries running over identical time periods in the past.

Just looking at the results:

Outer query	Time picker = Last Week	152,286,377 records
Inner query	Query time params: earliest=-w@w latest=@w	144,081,130 records

[niketn]

@ShagVT Seems with the append command like you are hitting the Subsearch limit configured in your instance . Because of which your inner search are dropping events. If you run the same searches (outer and inner) for -1d@d  to @d, I would hope that it will work as there would be events less than configured maxout.

You can definitely open Job Inspector to check and confirm this. 
For example:

info : [subsearch]: Subsearch produced ###### results, truncating to maxout 50000.

PS: I have picked 50k as maxout because that is the default limit. You can have something else.

You can configure maxout in the limits.conf. Refer to documentation: https://docs.splunk.com/Documentation/Splunk/latest/Search/Aboutsubsearches#Subsearch_performance_co...

However, you should rethink whether you really need to use a subsearch. If you are querying same data twice, for the same timeframe, you are possibly doing something wrong.

Can you perform a stats or a multisearch in worst case?

If you need further assistance you may have to provide us with further details about your SPL and use case so that community can assist you better.

[ShagVT]

Well, my goal is to report over different time periods.  I just wanted to check that the dashboard was working correctly so I tried setting the time periods to the same thing.  I posted about this approach in two other tickets as well:

https://community.splunk.com/t5/Splunk-Search/Compare-percentages-with-a-week-ago/m-p/513807#M144204

https://community.splunk.com/t5/Splunk-Search/Using-a-different-time-picker-on-an-inner-query/m-p/51...

What I'm trying to do doesn't seem like it would be such an edge case of functionality, but maybe it is.  I think the multisearch is ultimately just going to make the root problem worse.

[niketn]

Did you find the root problem yet? Did you get the subsearch limit error in the job inspector when you see different results. If you have subsearch limit impacting your result and you can write purely streaming search for two searches of a multisearch it will not be impacted by subsearch limitation for which you can read the documentation.

However like said before, maybe stats is your better option. Please let us know your SPL and some mock data would be useful as well. Kindly anonymize any sesitive info before posting.

[ShagVT]

@niketn - I don't think subsearch limit is the problem.  When I add up the number of rows my query is returning when I have it set for Previous Month for example (earliest=-mon@mon latest=@mon) I see it return  186,180,103 rows ... I'm certainly not running into any 50k limit.  That's because my inner query is already running stats to produce the basic numbers coming out, so it's only returning ~15 rows of already aggregated data.

Even so, the numbers are very different.  It counts 186M rows, while using the "Previous Month" time picker on the outer query returns 1,180,519,727 ... 10x as many. 

Here's the strangest part.  If I test looking at yesterday (Yesterday in the time picker, earliest=-d@d latest=@d for the inner query) the results are absolutely identical.  So it seems like my dashboard is more or less correct.  So I don't understand why the exact same logic doesn't work when i use a longer time period.

[niketn]

@ShagVT Please finish this ask to first to confirm....when the counts are not matching check job inspector whether you have any error/warning message or not.

Based on description 1 day works, 1 week does not, 1 month does not. It implies it has to do with events dropping. You think it is not subsearch limit so what limit have you configured? And event details for when it works vs when it does not.

[ITWhisperer]

Exactly what did you pick from the time picker to select "last week"?

[ShagVT]

Presets > Relative > Previous Week

[ITWhisperer]

Previous week snaps to beginning of day, whereas your other time period snaps to the beginning of the week. Good news though, it will be correct 1 day in 7 🙂

[ShagVT]

I don't know about that.  I ran this just now.

<basequery> 
| stats min(_time) as min, max(_time) as max, count
| eval max=strftime(max, "%Y/%m/%d %T.%3Q")
| eval min=strftime(min, "%Y/%m/%d %T.%3Q")
| table min max count

Results with "previous week" time picker	2020/08/23 00:00:00.078	2020/08/29 23:59:59.999	249617143
Results with "earliest=-w@w latest=@w"	2020/08/23 00:00:00.000	2020/08/29 23:59:59.999	263824202

So it's not a matter of the dates not lining up ... the dates appear to lineup *almost* precisely (just a difference of 0.078 seconds in the min).  I don't understand why the exact same query would produce such different results in count.