topic Re: Dissimilar results between sid & normal search in Splunk Search

Dissimilar results between sid & normal search

Naren26 — Tue, 03 Jul 2018 12:06:01 GMT

I am facing a weird issue with sid. I have a saved sid with yesterday's (00:00 to 23:59) data, which is showing a dip in my messages during yesterday evening. But, for the same query (since there is no change in the query, i believe I don't need to provide my query here) and for the same time frame, if I try to run as a new search, I do not see the dip.

Since, both are having same query & timeframe, how the results would change? Is there any difference in this? Someone please explain.

Re: Dissimilar results between sid & normal search

janispelss — Tue, 03 Jul 2018 14:20:07 GMT

Could it be that some of the data in that time range was indexed after you completed your original search, the one with the sid you're using?

Re: Dissimilar results between sid & normal search

DalJeanis — Tue, 29 Sep 2020 20:19:10 GMT

Try adding this to your later search

| where _indextime <= relative_time(now(),"@d")

and see if they go away.

If you know the exact time of your prior search, you could substitute an epoch-format version of that in the test.

Re: Dissimilar results between sid & normal search

Naren26 — Wed, 04 Jul 2018 13:19:42 GMT

It seems so!! Whatever I modify in that sid, there is no dip. May be caused by delay indexing??

Re: Dissimilar results between sid & normal search

janispelss — Wed, 04 Jul 2018 14:07:27 GMT

That does seem the most likely cause. The sid returns the exact results from when the search was originally run, apparently with incomplete data. Any new searches however have all the data available.

Re: Dissimilar results between sid & normal search

woodcock — Wed, 04 Jul 2018 16:26:39 GMT

You are missing something there, @daljeanis.

Re: Dissimilar results between sid & normal search

woodcock — Wed, 04 Jul 2018 16:29:55 GMT

Like @daljeanis implied, the problem is probably latency in your event delivery pipeline such that when you search for events way later, you see more of them because you gave them all enough time to get indexed and searchable. You can check latency either with the Meta W00t app or with a search like this:

| tstats max(_indextime) AS _indextime WHERE index=YourIndexHere AND sourcetype=YourSourcetypeHere BY _time host
| eval maxLatencySeconds = (_indextime - _time)
| stats avg(maxLatencySeconds) BY host