Why does expanding time range on search cause results to disappear?

[fvegdom]

When I run the following search with a time range restricted to a single day (9th of January)

index=main sourcetype=transaction application=a component=b customerCode=c Type=d messageType=e

I get a single result: (both search and result obfuscated for security reasons)

2017-01-09T13:36:56.109+0100    TRAN    b53a13ca-e1bc-4e64-964c-09c4714ba40e    custom-operations   process-engine  127.0.1.1   type:d|customerCode:c|bytesAllocated:15692632|executorUtilPct:0.0|Result:SUCCESS|messageType:e|traceId:1d250ddaa9b5ead479d2b7699b127c44

when I change the time range on this to run from the 8th to the 10th of January

I suddenly get No results found.

how can this happen?

[DalJeanis · Accepted solution]

At this point, I have to believe it is some sort of corrupted index problem . The multi-day search is using a search method that is somehow skipping data.

Use job inspector to take a look at each search and find out what is different between them, The multi-day search is either using a different index or not using an index while the single-day is going the other way. Something along those lines.

[fvegdom]

Thank you ill investigate that

[DalJeanis]

Just pulled that one out of the air, but my stats on total swags are about 80% hits. If I was TRYING to make this problem occur, then I'd just mess up an index. Voila! Problem created.

[fvegdom]

@DalJeanis

Ok, I have run both searches again and inspected te jobs, but I really don't know what I am looking for, the jobs look mostly the same to me.

the timeline view shows

when I go throug search.log, both searches mention the following:
Storing only 1000 events per timeline buckets due to limits.conf max_events_per_bucket setting.

is that normal?

the failing search shows more buckets in the timeline view, but that's not unexpected. There aren't any errors in either search. How can I see which index either job is using?

[DalJeanis]

1) Sorry about the delay. There is a graphic with line bars of how much machine time was used on each part of the search, in order. Compare the order of the search parts, and see if any are different, in different orders, or something like that.

2) The events per bucket worries me, but can't be particularly relevant to this issue, since that one peekaboo event would be in a bucket by itself and thus never be dropped.

[somesoni2]

Do you see this behavior for any other day or just for Jan 9?

[fvegdom]

after some investigation I find it for other days as well

for example if I run it for only the 12th I get 49 results, then when I expand from 11 to 13 I get 40 results, 15 of which are on the 12th, this is with the same search.