Splunk Search

Distributed search scheduled alerts on SH

Mag2sub
Path Finder

We have an indexer indexing events with _time 5 hours head and we have Distributed search from SH which looks at _index time earliest and latest 10 minutes...although events with _time + 5 hours and matching index time exist ..they dont show up in Splunk SH scheduled searches ? why ?

Does the scheduler (SH) introduce some filter when they run to prevent them from searching events that have timestamps later than the local runtimes of the queries
Kindly clarify

Tags (1)
0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Yeah, but _index_earliest and _index_latest are added to the time range as AND-condition, they don't replace it. Here's an example:

I've just started the Splunk on my Laptop, it's 9am... in the past hour it has indexed data from this morning, plus a bit of leftovers from yesterday. I'm running two searches based on this string:

index=_internal _index_earliest=-h _index_latest=now

If I run that with a time range (timestamp, not index time) of -24h@h to now I get events from yesterday that were indexed this morning, if I run that with a time range (timestamp, not index time) of -4h@m to now I don't get those.

To get around this for events from the future, set the time range's latest setting to something in the future, for example +2d.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

Yeah, but _index_earliest and _index_latest are added to the time range as AND-condition, they don't replace it. Here's an example:

I've just started the Splunk on my Laptop, it's 9am... in the past hour it has indexed data from this morning, plus a bit of leftovers from yesterday. I'm running two searches based on this string:

index=_internal _index_earliest=-h _index_latest=now

If I run that with a time range (timestamp, not index time) of -24h@h to now I get events from yesterday that were indexed this morning, if I run that with a time range (timestamp, not index time) of -4h@m to now I don't get those.

To get around this for events from the future, set the time range's latest setting to something in the future, for example +2d.

martin_mueller
SplunkTrust
SplunkTrust

The time range never appears in the lispy debug messages.

Try choosing latest=+10y to effectively disable the latest time range boundary - come back once you've tried that and report on your results.

0 Karma

Mag2sub
Path Finder

..i have searched the search.log Lispy query search parser of this search for these _indextime "AND" all _time cannot seem to find it ...is the final query dispatched available to see in any logs..i keep seeing only indextime..

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

All filters are linked by AND, especially the time range. Just set latest to +10y.

0 Karma

Mag2sub
Path Finder

On a concept level if we have inline search time range of past 5 minutes of indextime and outside timerange of past 10 min..so will it AND and always give both last 5 indextime and last 10 _time..making our key purpose ineffective
I did not get the precedence or how do 2 time ranges operate ?

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Use +10y then, that's most likely beyond your MAX_DAYS_HENCE setting anyway.

0 Karma

Mag2sub
Path Finder

Is there anyway we can just search using _indextime and not use _time

0 Karma

Mag2sub
Path Finder

Ok the idea is we dont know all device timezone and timestamps...how can we be sure of +2d ...it can be a year head ..irrespective we use indextime in seach query

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

If you specify _index_earliest inline then you're not overriding any earliest set elsewhere. Both filters can coexist and be applied.

As for "no _time", every event in Splunk should have _time, even if it's just derived from the index time.

If the timerange is blank then it should use "all time", which is earliest=0 latest=now.

Have you tried what I suggested several times, setting latest as a time in the future, e.g. +2d?

0 Karma

Mag2sub
Path Finder

Also note i use scheduled search and timerange is blank excepting for inline search timerange based on _index_time

0 Karma

Mag2sub
Path Finder

Hmm... inline search timerange is supposed to override Timerange specified outside ...i saw that documented somewhere ..also in our case the timerange outside our search is blank...ie we have no _time

0 Karma

Mag2sub
Path Finder

I believe you can do _index_earliest and latest based search instead of _time and thats what was used

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Isn't every search based on _time, regardless of other filters?

0 Karma

Mag2sub
Path Finder

I guess im trying to say the search query is based on _indextime and not _time...so that should not be the case although _time is +5 hrs ahead

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

If your earliest and latest filter for ten minutes then you won't see events a day into the future.

0 Karma

Mag2sub
Path Finder

Distributed search from SH which looks at _index time earliest and latest 10 minutes..as in above ...thanks

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

What time range is the scheduled search using?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...