Splunk Enterprise

Searches are not using earliest and latest time modifiers

TheJagoff
Communicator

Hello, and I have another weird issue:
When I execute a search on a SHC in the Search and Reporting App, getting data from 2025-02-27
index=test earliest=-7d@d latest=-6d@d
I get zero events
When I execute the search WITHOUT the earliest and latest time modifiers and use the Time Picker in the UI which results in "during Thu, Feb 27, 2025"
I get around 167,153 results

Specifying the time range with earliest and latest time modifiers is NOT giving me the "Your timerange was substituted based on your search string".

If I use tstats, I get the correct number of events, the correct date, and the message "Your timerange was substituted based on your search string" is present

| tstats count where index=test earliest=-7d@d latest=-6d@d by _time span=d

I also made index=test earliest=-7d@d latest=-6d@d a saved search which executes every 10 minutes - zero events.

Another bit of weirdness:
If I run that search, and specify "All time", it will pull events ONLY for 2025-02-27. Nothing for other dates, and it has 12 months of events, populated for every day. So, it looks at both the time qualifiers and the time picker under that scenario.

Any ideas what might be causing this? (I have several standalone searchheads that are working fine)

Labels (1)
0 Karma
1 Solution

TheJagoff
Communicator

Found the issue:
We built a standalone SH and copied the $SPLUNK_HOME/etc/apps directory from the SHC to it. 
Started removing apps on the test server, one at a time, and when we removed one of the Apps and restarted., the searches started to work again.
One of our crew found the following in the app the was just removed:

[source::stream:Gigamon]
EVAL-_time = strptime('timestamp', "%Y-%m-%dT%H:%M:%S,%N")

This seems to be the issue. We went back to the SHC and specified a source without removing anything and it pulled data. Not really clear on why that would make a difference, but it does.

The main takeaway from this is that a configuration change that had an effect on _time caused this issue.

View solution in original post

TheJagoff
Communicator

Found the issue:
We built a standalone SH and copied the $SPLUNK_HOME/etc/apps directory from the SHC to it. 
Started removing apps on the test server, one at a time, and when we removed one of the Apps and restarted., the searches started to work again.
One of our crew found the following in the app the was just removed:

[source::stream:Gigamon]
EVAL-_time = strptime('timestamp', "%Y-%m-%dT%H:%M:%S,%N")

This seems to be the issue. We went back to the SHC and specified a source without removing anything and it pulled data. Not really clear on why that would make a difference, but it does.

The main takeaway from this is that a configuration change that had an effect on _time caused this issue.

Get Updates on the Splunk Community!

Holistic Visibility and Effective Alerting Across IT and OT Assets

Instead of effective and unified solutions, they’re left with tool fatigue, disjointed alerts and siloed ...

SOC Modernization: How Automation and Splunk SOAR are Shaping the Next-Gen Security ...

Security automation is no longer a luxury but a necessity. Join us to learn how Splunk ES and SOAR empower ...

Ask It, Fix It: Faster Investigations with AI Assistant in Observability Cloud

  Join us in this Tech Talk and learn about the recently launched AI Assistant in Observability Cloud. With ...