Splunk Enterprise

Searches are not using earliest and latest time modifiers

TheJagoff
Communicator

Hello, and I have another weird issue:
When I execute a search on a SHC in the Search and Reporting App, getting data from 2025-02-27
index=test earliest=-7d@d latest=-6d@d
I get zero events
When I execute the search WITHOUT the earliest and latest time modifiers and use the Time Picker in the UI which results in "during Thu, Feb 27, 2025"
I get around 167,153 results

Specifying the time range with earliest and latest time modifiers is NOT giving me the "Your timerange was substituted based on your search string".

If I use tstats, I get the correct number of events, the correct date, and the message "Your timerange was substituted based on your search string" is present

| tstats count where index=test earliest=-7d@d latest=-6d@d by _time span=d

I also made index=test earliest=-7d@d latest=-6d@d a saved search which executes every 10 minutes - zero events.

Another bit of weirdness:
If I run that search, and specify "All time", it will pull events ONLY for 2025-02-27. Nothing for other dates, and it has 12 months of events, populated for every day. So, it looks at both the time qualifiers and the time picker under that scenario.

Any ideas what might be causing this? (I have several standalone searchheads that are working fine)

Labels (1)
0 Karma
1 Solution

TheJagoff
Communicator

Found the issue:
We built a standalone SH and copied the $SPLUNK_HOME/etc/apps directory from the SHC to it. 
Started removing apps on the test server, one at a time, and when we removed one of the Apps and restarted., the searches started to work again.
One of our crew found the following in the app the was just removed:

[source::stream:Gigamon]
EVAL-_time = strptime('timestamp', "%Y-%m-%dT%H:%M:%S,%N")

This seems to be the issue. We went back to the SHC and specified a source without removing anything and it pulled data. Not really clear on why that would make a difference, but it does.

The main takeaway from this is that a configuration change that had an effect on _time caused this issue.

View solution in original post

TheJagoff
Communicator

Found the issue:
We built a standalone SH and copied the $SPLUNK_HOME/etc/apps directory from the SHC to it. 
Started removing apps on the test server, one at a time, and when we removed one of the Apps and restarted., the searches started to work again.
One of our crew found the following in the app the was just removed:

[source::stream:Gigamon]
EVAL-_time = strptime('timestamp', "%Y-%m-%dT%H:%M:%S,%N")

This seems to be the issue. We went back to the SHC and specified a source without removing anything and it pulled data. Not really clear on why that would make a difference, but it does.

The main takeaway from this is that a configuration change that had an effect on _time caused this issue.

Get Updates on the Splunk Community!

New This Month - Splunk Observability updates and improvements for faster ...

What’s New? This month, we’re delivering several enhancements across Splunk Observability Cloud for faster and ...

What's New in Splunk Cloud Platform 9.3.2411?

Hey Splunky People! We are excited to share the latest updates in Splunk Cloud Platform 9.3.2411. This release ...

Buttercup Games: Further Dashboarding Techniques (Part 6)

This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the ...