Splunk Search

Increasing Time Range hides Data

amat
Explorer

Hey y'all,

So I am seeing a very unique and strange behavior from Splunk. I noticed an issue where a Splunk search returned an event if the time range was narrow, but would not return the same event if the time range was broader.

For example, the following Splunk SPL would return 1 event when the time range was set to May 2nd to May 3rd.

index=firewalls  dest_ip=124.65.193.752 src_ip=10.42.68.21

However, the same SPL would return 0 events when the time range picker was set to "Last 90 days"

index=firewalls   dest_ip=124.65.193.752 src_ip=10.42.68.21

The last 30 days from today would essentially cover March 15th to June 15th (Today).

We have a ton of Firewall logs so the only thing I can think of is that Splunk accidentally skips over the event due to the sheer volume of data....however no warning is thrown after the search is run. Splunk throws no warning or issues under the Job tab.

Any ideas would greatly be appreciated!

0 Karma

jnudell_2
Builder

Is that consistent behavior? Every time you search for the last 90 days you don't see that event? I do agree with @Vijeta 's comment to try using the earliest=@d-90d latest=@d time range in the search itself to see if you return different results.

0 Karma

amat
Explorer

@jnudell No it only appeared in this one case. I havent seen this happen before. -90 seems to fix it but i dont understand why.

0 Karma

ragedsparrow
Contributor

Have you inspected the job after it's run with the time picker to see if anything might be happening to it? It seems odd that if you choose "Last 90 Days" in the time picker you would get no results, but if you do earliest=-90d you don't have that issue. You may need to engage support to troubleshoot this.

0 Karma

amat
Explorer

@jnudell_2 @ragedsparrow i misspoke. I was looking at the wrong search.

So earliest=-90d doesnt work. The time range picker seems to take priority when do a search.

0 Karma

jnudell_2
Builder

earliest & latest statements in the search query ALWAYS take precedence over the time picker from the UI.

0 Karma

Vijeta
Influencer

@amat - Try using earliest=-90@d in your query.

amat
Explorer

@vijeta I forgot to mention that when i do a 90 day search, the event tab shows " Events(1) " , meaning it found 1 event. However the results at the bottom says "No results found. Try expanding the time range.".

I am confused why its showing that it found 1 event but doesnt display it

0 Karma

amat
Explorer

@Vijeta Yes this seems to fix it...any idea why?

0 Karma

Vijeta
Influencer

@amat Are you logged in as admin? If not may be its some restriction on your role.

0 Karma

amat
Explorer

I am an admin. I dont have any restriction on my role

0 Karma

Vijeta
Influencer

That's kind of strange.

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...