Increasing Time Range hides Data

[amat]

Hey y'all,

So I am seeing a very unique and strange behavior from Splunk. I noticed an issue where a Splunk search returned an event if the time range was narrow, but would not return the same event if the time range was broader.

For example, the following Splunk SPL would return 1 event when the time range was set to May 2nd to May 3rd.

index=firewalls  dest_ip=124.65.193.752 src_ip=10.42.68.21

However, the same SPL would return 0 events when the time range picker was set to "Last 90 days"

index=firewalls   dest_ip=124.65.193.752 src_ip=10.42.68.21

The last 30 days from today would essentially cover March 15th to June 15th (Today).

We have a ton of Firewall logs so the only thing I can think of is that Splunk accidentally skips over the event due to the sheer volume of data....however no warning is thrown after the search is run. Splunk throws no warning or issues under the Job tab.

Any ideas would greatly be appreciated!

[jnudell_2]

Is that consistent behavior? Every time you search for the last 90 days you don't see that event? I do agree with @Vijeta 's comment to try using the earliest=@d-90d latest=@d time range in the search itself to see if you return different results.

[amat]

@jnudell No it only appeared in this one case. I havent seen this happen before. -90 seems to fix it but i dont understand why.

[ragedsparrow]

Have you inspected the job after it's run with the time picker to see if anything might be happening to it? It seems odd that if you choose "Last 90 Days" in the time picker you would get no results, but if you do earliest=-90d you don't have that issue. You may need to engage support to troubleshoot this.

[amat]

@jnudell_2 @ragedsparrow i misspoke. I was looking at the wrong search.

So earliest=-90d doesnt work. The time range picker seems to take priority when do a search.

[jnudell_2]

earliest & latest statements in the search query ALWAYS take precedence over the time picker from the UI.

[Vijeta]

@amat - Try using earliest=-90@d in your query.

[amat]

@vijeta I forgot to mention that when i do a 90 day search, the event tab shows " Events(1) " , meaning it found 1 event. However the results at the bottom says "No results found. Try expanding the time range.".

I am confused why its showing that it found 1 event but doesnt display it

[amat]

@Vijeta Yes this seems to fix it...any idea why?

[Vijeta]

@amat Are you logged in as admin? If not may be its some restriction on your role.

[amat]

I am an admin. I dont have any restriction on my role

[Vijeta]

That's kind of strange.