I am facing a weird issue with sid. I have a saved sid with yesterday's (00:00 to 23:59) data, which is showing a dip in my messages during yesterday evening. But, for the same query (since there is no change in the query, i believe I don't need to provide my query here) and for the same time frame, if I try to run as a new search, I do not see the dip.
Since, both are having same query & timeframe, how the results would change? Is there any difference in this? Someone please explain.
Could it be that some of the data in that time range was indexed after you completed your original search, the one with the sid you're using?
That does seem the most likely cause. The sid returns the exact results from when the search was originally run, apparently with incomplete data. Any new searches however have all the data available.
Try adding this to your later search
| where indextime <= relativetime(now(),"@d")
and see if they go away.
If you know the exact time of your prior search, you could substitute an epoch-format version of that in the test.
Like @daljeanis implied, the problem is probably latency in your event delivery pipeline such that when you search for events way later, you see more of them because you gave them all enough time to get indexed and searchable. You can check latency either with the
Meta W00t app or with a search like this:
| tstats max(_indextime) AS _indextime WHERE index=YourIndexHere AND sourcetype=YourSourcetypeHere BY _time host | eval maxLatencySeconds = (_indextime - _time) | stats avg(maxLatencySeconds) BY host