topic Re: search start time in Splunk Search

search start time

guilhem — Mon, 28 Sep 2020 13:34:00 GMT

HI,

I would like to know if it's possible in the earliest / latest fields of a search to have something like:

index=myindex earliest="the time this search has started"-X seconds latest="the time this search has started" -Y seconds

Is it possible?

The aim is to run a saved search that populates a summary index, but I need to backfill this index with a search that contains earliest=-20h latest=-10h, but running the search as it is with the fill_summary_index.py command line returns no results because events I need to backfill occurs a long time before -20h (I need to backfill 4 month of datas).

I think I can have an eval statement that would compute what I want, and then just use where, but it would be very inefficient time wise.

Thanks,

EDIT

Thanks to the answers, I found that I can use earliest= [some search | return result] to populate the earliest field to look for data when I want. Problem is that I can't find a way to say:

earliest=["search that returns the starting scheduled time of this saved search"]

The keyword 'now' returns the starting time of the search when put inside the earliest field, but it's not what I want, I would like to have the starting SCHEDULED time, not the actual time I run the search.

I don't know if it's the right way to do it, and if there is another way, I would gladly try it.

Re: search start time

martin_mueller — Thu, 21 Mar 2013 13:00:07 GMT

If you can express your time fields using eval you can do a subsearch for each:

index=myindex earliest=[some search | eval earliest=something | return $earliest] latest=[some search | eval latest=something | return $latest] | ...

Re: search start time

guilhem — Thu, 21 Mar 2013 13:41:15 GMT

THAT is awesome, didn't know you could run subsearches after an '=' !!!

Is it possible to run a subsearch like this: 'eval=[some subsearch]' ? It looks increadibly powerfull and will solve many performance problem that I have. Gonna test it right away.

Re: search start time

guilhem — Mon, 28 Sep 2020 13:34:08 GMT

I have tried it but I couldn't make it works:

index=_internal earliest=[ search index=_internal | head 1 | return "-24h"]
, also tried:

index=_internal earliest=[ search index=_internal | head 1 | eval test="-24h" | return test]

but it always returns:

Error in 'search' command: Unable to parse the search: Comparator '=' has an invalid term on the left hand side.

Re: search start time

guilhem — Mon, 28 Sep 2020 13:34:11 GMT

Peoblem solved, in fact result contained "test=-24h" and not only "-24h" as I was expected.

The correct search is then:

index=_internal [search index=_internal | head 1 | eval earliest="-24h" | return earliest]

which works wonderfully.

Many thanks for the help.

Re: search start time

guilhem — Thu, 21 Mar 2013 14:19:08 GMT

I just need to figure out how to get the time the search was scheduled, instead of the "now" time.

Re: search start time

martin_mueller — Thu, 21 Mar 2013 14:20:01 GMT

Your first attempt can be made to work like this:

index=_internal earliest=[ stats count | eval test="-24h" | return $test]

The dollar sign changes the behaviour of return, returning only the value itself instead of key=value as usual.

Re: search start time

guilhem — Thu, 21 Mar 2013 15:47:10 GMT

nice! Another tip I didn't know.

I am still trying to find a way to get the search scheduled time start, but I didn't find it yet. Does scheduled search have a special field containing their scheduled time?

Re: search start time

martin_mueller — Thu, 21 Mar 2013 15:52:20 GMT

Isn't the time the search was run equal to now from the search's point of view?

Re: search start time

guilhem — Mon, 28 Sep 2020 13:34:17 GMT

Thank you very much martin for the help. I finally found the rest of the solution from here, use | addinfo, and info_min_time to retrieve the starting time of the search. So the final answer is (if you want earliest to start 20 hours before the scheduled time of the search eg):

index=_internal earliest=[ search index=_internal | head 1| addinfo | eval test=info_min_time-20*3600 | return $test]

EDIT

simplified, optimized, cleaned version:

index=_internal earliest=[ stats count | addinfo | eval test=relative_time(info_min_time, "-20h") | return $test]

Re: search start time

martin_mueller — Thu, 21 Mar 2013 16:18:30 GMT

Since you're not actually using any data from _internal you could replace that with a call to stats or something else that doesn't cause any data to be loaded:

index=_internal earliest=[ stats count | addinfo | eval test=relative_time(info_min_time, "-20h") | return $test]

Additionally, you can let Splunk's relative time syntax do the time fiddling for you in case it gets more complicated.

Re: search start time

guilhem — Thu, 21 Mar 2013 16:24:59 GMT

Thank you very much, I have learned a lots of very valuable things on splunk today. It's been a loooong time since I want to find a way to have something evalued whithout pulling datas:

doesn't work, so I used to play with index=X | head 1 so it was really fast. Yours (using stats) is even better.

Never used relative_time before. It's really clean and less "hacky" than what I did. I update the answer