topic Re: Distributed search scheduled alerts on SH in Splunk Search

Distributed search scheduled alerts on SH

Mag2sub — Mon, 12 May 2014 08:49:15 GMT

We have an indexer indexing events with _time 5 hours head and we have Distributed search from SH which looks at _index time earliest and latest 10 minutes...although events with _time + 5 hours and matching index time exist ..they dont show up in Splunk SH scheduled searches ? why ?

Does the scheduler (SH) introduce some filter when they run to prevent them from searching events that have timestamps later than the local runtimes of the queries
Kindly clarify

Re: Distributed search scheduled alerts on SH

martin_mueller — Mon, 12 May 2014 21:29:14 GMT

What time range is the scheduled search using?

Re: Distributed search scheduled alerts on SH

Mag2sub — Tue, 13 May 2014 03:29:52 GMT

Distributed search from SH which looks at _index time earliest and latest 10 minutes..as in above ...thanks

Re: Distributed search scheduled alerts on SH

martin_mueller — Tue, 13 May 2014 08:01:40 GMT

If your earliest and latest filter for ten minutes then you won't see events a day into the future.

Re: Distributed search scheduled alerts on SH

Mag2sub — Wed, 14 May 2014 06:23:15 GMT

I guess im trying to say the search query is based on _indextime and not _time...so that should not be the case although _time is +5 hrs ahead

Re: Distributed search scheduled alerts on SH

martin_mueller — Wed, 14 May 2014 07:05:43 GMT

Isn't every search based on _time, regardless of other filters?

Re: Distributed search scheduled alerts on SH

Mag2sub — Mon, 28 Sep 2020 16:37:05 GMT

I believe you can do _index_earliest and latest based search instead of _time and thats what was used

Re: Distributed search scheduled alerts on SH

martin_mueller — Thu, 15 May 2014 07:13:42 GMT

Yeah, but _index_earliest and _index_latest are added to the time range as AND-condition, they don't replace it. Here's an example:

I've just started the Splunk on my Laptop, it's 9am... in the past hour it has indexed data from this morning, plus a bit of leftovers from yesterday. I'm running two searches based on this string:

index=_internal _index_earliest=-h _index_latest=now

If I run that with a time range (timestamp, not index time) of -24h@h to now I get events from yesterday that were indexed this morning, if I run that with a time range (timestamp, not index time) of -4h@m to now I don't get those.

To get around this for events from the future, set the time range's latest setting to something in the future, for example +2d.

Re: Distributed search scheduled alerts on SH

Mag2sub — Thu, 15 May 2014 08:56:10 GMT

Hmm... inline search timerange is supposed to override Timerange specified outside ...i saw that documented somewhere ..also in our case the timerange outside our search is blank...ie we have no _time

Re: Distributed search scheduled alerts on SH

Mag2sub — Mon, 28 Sep 2020 16:37:25 GMT

Also note i use scheduled search and timerange is blank excepting for inline search timerange based on _index_time

Re: Distributed search scheduled alerts on SH

martin_mueller — Thu, 15 May 2014 21:53:23 GMT

If you specify _index_earliest inline then you're not overriding any earliest set elsewhere. Both filters can coexist and be applied.

As for "no _time", every event in Splunk should have _time, even if it's just derived from the index time.

If the timerange is blank then it should use "all time", which is earliest=0 latest=now.

Have you tried what I suggested several times, setting latest as a time in the future, e.g. +2d?

Re: Distributed search scheduled alerts on SH

Mag2sub — Fri, 16 May 2014 09:44:40 GMT

Ok the idea is we dont know all device timezone and timestamps...how can we be sure of +2d ...it can be a year head ..irrespective we use indextime in seach query

Re: Distributed search scheduled alerts on SH

Mag2sub — Fri, 16 May 2014 09:45:15 GMT

Is there anyway we can just search using _indextime and not use _time

Re: Distributed search scheduled alerts on SH

martin_mueller — Fri, 16 May 2014 10:38:12 GMT

Use +10y then, that's most likely beyond your MAX_DAYS_HENCE setting anyway.

Re: Distributed search scheduled alerts on SH

Mag2sub — Sat, 17 May 2014 03:41:40 GMT

On a concept level if we have inline search time range of past 5 minutes of indextime and outside timerange of past 10 min..so will it AND and always give both last 5 indextime and last 10 _time..making our key purpose ineffective
I did not get the precedence or how do 2 time ranges operate ?

Re: Distributed search scheduled alerts on SH

martin_mueller — Sat, 17 May 2014 10:46:00 GMT

All filters are linked by AND, especially the time range. Just set latest to +10y.

Re: Distributed search scheduled alerts on SH

Mag2sub — Sun, 18 May 2014 16:34:43 GMT

..i have searched the search.log Lispy query search parser of this search for these _indextime "AND" all _time cannot seem to find it ...is the final query dispatched available to see in any logs..i keep seeing only indextime..

Re: Distributed search scheduled alerts on SH

martin_mueller — Sun, 18 May 2014 22:16:22 GMT

The time range never appears in the lispy debug messages.

Try choosing latest=+10y to effectively disable the latest time range boundary - come back once you've tried that and report on your results.