Splunk Search

Why does expanding time range on search cause results to disappear?

fvegdom
Path Finder

When I run the following search with a time range restricted to a single day (9th of January)

index=main sourcetype=transaction application=a component=b customerCode=c Type=d messageType=e

I get a single result: (both search and result obfuscated for security reasons)

2017-01-09T13:36:56.109+0100    TRAN    b53a13ca-e1bc-4e64-964c-09c4714ba40e    custom-operations   process-engine  127.0.1.1   type:d|customerCode:c|bytesAllocated:15692632|executorUtilPct:0.0|Result:SUCCESS|messageType:e|traceId:1d250ddaa9b5ead479d2b7699b127c44

when I change the time range on this to run from the 8th to the 10th of January

I suddenly get No results found.

how can this happen?

0 Karma
1 Solution

DalJeanis
SplunkTrust
SplunkTrust

At this point, I have to believe it is some sort of corrupted index problem . The multi-day search is using a search method that is somehow skipping data.

Use job inspector to take a look at each search and find out what is different between them, The multi-day search is either using a different index or not using an index while the single-day is going the other way. Something along those lines.

View solution in original post

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

At this point, I have to believe it is some sort of corrupted index problem . The multi-day search is using a search method that is somehow skipping data.

Use job inspector to take a look at each search and find out what is different between them, The multi-day search is either using a different index or not using an index while the single-day is going the other way. Something along those lines.

0 Karma

fvegdom
Path Finder

Thank you ill investigate that

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Just pulled that one out of the air, but my stats on total swags are about 80% hits. If I was TRYING to make this problem occur, then I'd just mess up an index. Voila! Problem created.

0 Karma

fvegdom
Path Finder

@DalJeanis

Ok, I have run both searches again and inspected te jobs, but I really don't know what I am looking for, the jobs look mostly the same to me.

the timeline view shows

when I go throug search.log, both searches mention the following:
Storing only 1000 events per timeline buckets due to limits.conf max_events_per_bucket setting.

is that normal?

the failing search shows more buckets in the timeline view, but that's not unexpected. There aren't any errors in either search. How can I see which index either job is using?

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

1) Sorry about the delay. There is a graphic with line bars of how much machine time was used on each part of the search, in order. Compare the order of the search parts, and see if any are different, in different orders, or something like that.

2) The events per bucket worries me, but can't be particularly relevant to this issue, since that one peekaboo event would be in a bucket by itself and thus never be dropped.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Do you see this behavior for any other day or just for Jan 9?

0 Karma

fvegdom
Path Finder

after some investigation I find it for other days as well

for example if I run it for only the 12th I get 49 results, then when I expand from 11 to 13 I get 40 results, 15 of which are on the 12th, this is with the same search.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...