Why are the search results changing days after running the same search?

[bsree]

We are periodically seeing instances where data that was previously indexed shows up differently.
The results I got for a sample search changed after some days when I ran the same search again for the same time period.

Kindly help me to figure out why is this happening.
This problem is only with a particular index, all others are working fine.

[evania]

Hi @bsree ,

Did you have a chance to check answers yet? If it worked, please resolve this post by approving it! If your problem is still not solved, keep us updated so that someone else can help you.

Thanks for posting!

[stcrispan]

martinpu - thanks for your help. We tried them out, and now there are even more questions...

Attempting these queries...the first one produced no results for a 24 hour period, but counting over 7 days, produced some numbers in the chart ranging from 1 up to 37. Are these in seconds, milliseconds, count of events which were delayed? Which ones?

The second query showed, over seven days, a minimum "rounded" of 0.0 (count 5) and maximum "rounded" of 222.0 (count 1). Again, this query reveals the "time difference delay", but in what measurement? Does this say that there were 5 events which had zero lag, and 1 which experienced 222 lag? 222 what? Seconds, milliseconds, minutes?

And, once we've got that definition, how does the indexing delay cause the charts produced for the same day to be different? I thought that the data, once recorded, was immutable. Seems like it would have to be a pretty sizable delay...

Again, appreciate the help, it's just that we've only been using it for about a year...

[niketn]

@bsree please add more details to your question. What do you mean by "shows up differently"? Does it increase? Does it decrease or is it random? What is the current query? Are you showing stats or is it raw data search? Do you have sub-searches in your query?What is the volume of data coming to index? Is it possible that the indexed data is getting purged?

[bsree]

Its random, It decreases as well as increases. my query is this { index="dss_tablet" host="SLATE501*" EventCode=1991 | timechart count as Occurrences }. previously when I searched there were more than 400 events, right now if i search there are only 255 events. I don't think indexed data is getting purged.

[martinpu]

Perhaps your events are arriving to the index with a delay?

try this an see when your events were indexed:

your query
|bucket _indextime bins=500 
|chart count by _indextime 

or to figure out the timedifference(delay of event happening vs reaching the index)

your query
|eval delay=_indextime-_time
| eval rounded=round(delay,1)
|chart count by rounded

This will only work if you have access to _indextime which you might not have access to by default,