Splunk Enterprise

Searches are not using earliest and latest time modifiers

TheJagoff
Communicator

Hello, and I have another weird issue:
When I execute a search on a SHC in the Search and Reporting App, getting data from 2025-02-27
index=test earliest=-7d@d latest=-6d@d
I get zero events
When I execute the search WITHOUT the earliest and latest time modifiers and use the Time Picker in the UI which results in "during Thu, Feb 27, 2025"
I get around 167,153 results

Specifying the time range with earliest and latest time modifiers is NOT giving me the "Your timerange was substituted based on your search string".

If I use tstats, I get the correct number of events, the correct date, and the message "Your timerange was substituted based on your search string" is present

| tstats count where index=test earliest=-7d@d latest=-6d@d by _time span=d

I also made index=test earliest=-7d@d latest=-6d@d a saved search which executes every 10 minutes - zero events.

Another bit of weirdness:
If I run that search, and specify "All time", it will pull events ONLY for 2025-02-27. Nothing for other dates, and it has 12 months of events, populated for every day. So, it looks at both the time qualifiers and the time picker under that scenario.

Any ideas what might be causing this? (I have several standalone searchheads that are working fine)

Labels (1)
0 Karma
1 Solution

TheJagoff
Communicator

Found the issue:
We built a standalone SH and copied the $SPLUNK_HOME/etc/apps directory from the SHC to it. 
Started removing apps on the test server, one at a time, and when we removed one of the Apps and restarted., the searches started to work again.
One of our crew found the following in the app the was just removed:

[source::stream:Gigamon]
EVAL-_time = strptime('timestamp', "%Y-%m-%dT%H:%M:%S,%N")

This seems to be the issue. We went back to the SHC and specified a source without removing anything and it pulled data. Not really clear on why that would make a difference, but it does.

The main takeaway from this is that a configuration change that had an effect on _time caused this issue.

View solution in original post

TheJagoff
Communicator

Found the issue:
We built a standalone SH and copied the $SPLUNK_HOME/etc/apps directory from the SHC to it. 
Started removing apps on the test server, one at a time, and when we removed one of the Apps and restarted., the searches started to work again.
One of our crew found the following in the app the was just removed:

[source::stream:Gigamon]
EVAL-_time = strptime('timestamp', "%Y-%m-%dT%H:%M:%S,%N")

This seems to be the issue. We went back to the SHC and specified a source without removing anything and it pulled data. Not really clear on why that would make a difference, but it does.

The main takeaway from this is that a configuration change that had an effect on _time caused this issue.

Get Updates on the Splunk Community!

CX Day is Coming!

Customer Experience (CX) Day is on October 7th!! We're so excited to bring back another day full of wonderful ...

Strengthen Your Future: A Look Back at Splunk 10 Innovations and .conf25 Highlights!

The Big One: Splunk 10 is Here!  The moment many of you have been waiting for has arrived! We are thrilled to ...

Now Offering the AI Assistant Usage Dashboard in Cloud Monitoring Console

Today, we’re excited to announce the release of a brand new AI assistant usage dashboard in Cloud Monitoring ...