Splunk Search
Highlighted

Dissimilar results between sid & normal search

Path Finder

I am facing a weird issue with sid. I have a saved sid with yesterday's (00:00 to 23:59) data, which is showing a dip in my messages during yesterday evening. But, for the same query (since there is no change in the query, i believe I don't need to provide my query here) and for the same time frame, if I try to run as a new search, I do not see the dip.

Since, both are having same query & timeframe, how the results would change? Is there any difference in this? Someone please explain.

0 Karma
Highlighted

Re: Dissimilar results between sid & normal search

Path Finder

Could it be that some of the data in that time range was indexed after you completed your original search, the one with the sid you're using?

Highlighted

Re: Dissimilar results between sid & normal search

Path Finder

It seems so!! Whatever I modify in that sid, there is no dip. May be caused by delay indexing??

0 Karma
Highlighted

Re: Dissimilar results between sid & normal search

Path Finder

That does seem the most likely cause. The sid returns the exact results from when the search was originally run, apparently with incomplete data. Any new searches however have all the data available.

0 Karma
Highlighted

Re: Dissimilar results between sid & normal search

SplunkTrust
SplunkTrust

Try adding this to your later search

| where indextime <= relativetime(now(),"@d")

and see if they go away.

If you know the exact time of your prior search, you could substitute an epoch-format version of that in the test.

0 Karma
Highlighted

Re: Dissimilar results between sid & normal search

Esteemed Legend

You are missing something there, @daljeanis.

0 Karma
Highlighted

Re: Dissimilar results between sid & normal search

Esteemed Legend

Like @daljeanis implied, the problem is probably latency in your event delivery pipeline such that when you search for events way later, you see more of them because you gave them all enough time to get indexed and searchable. You can check latency either with the Meta W00t app or with a search like this:

| tstats max(_indextime) AS _indextime WHERE index=YourIndexHere AND sourcetype=YourSourcetypeHere BY _time host
| eval maxLatencySeconds = (_indextime - _time)
| stats avg(maxLatencySeconds) BY host
0 Karma