I would like to know if it's possible in the earliest / latest fields of a search to have something like:
index=myindex earliest="the time this search has started"-X seconds latest="the time this search has started" -Y seconds
Is it possible?
The aim is to run a saved search that populates a summary index, but I need to backfill this index with a search that contains earliest=-20h latest=-10h, but running the search as it is with the fillsummaryindex.py command line returns no results because events I need to backfill occurs a long time before -20h (I need to backfill 4 month of datas).
I think I can have an eval statement that would compute what I want, and then just use where, but it would be very inefficient time wise.
Thanks to the answers, I found that I can use earliest= [some search | return result] to populate the earliest field to look for data when I want. Problem is that I can't find a way to say:
earliest=["search that returns the starting scheduled time of this saved search"]
The keyword 'now' returns the starting time of the search when put inside the earliest field, but it's not what I want, I would like to have the starting SCHEDULED time, not the actual time I run the search.
I don't know if it's the right way to do it, and if there is another way, I would gladly try it.
If you can express your time fields using eval you can do a subsearch for each:
index=myindex earliest=[some search | eval earliest=something | return $earliest] latest=[some search | eval latest=something | return $latest] | ...
THAT is awesome, didn't know you could run subsearches after an '=' !!!
Is it possible to run a subsearch like this: 'eval=[some subsearch]' ? It looks increadibly powerfull and will solve many performance problem that I have. Gonna test it right away.
I have tried it but I couldn't make it works:
index=internal earliest=[ search index=internal | head 1 | return "-24h"]
, also tried:
index=internal earliest=[ search index=internal | head 1 | eval test="-24h" | return test]
but it always returns:
Error in 'search' command: Unable to parse the search: Comparator '=' has an invalid term on the left hand side.
Peoblem solved, in fact result contained "test=-24h" and not only "-24h" as I was expected.
The correct search is then:
index=internal [search index=internal | head 1 | eval earliest="-24h" | return earliest]
which works wonderfully.
Many thanks for the help.
Your first attempt can be made to work like this:
index=_internal earliest=[ stats count | eval test="-24h" | return $test]
The dollar sign changes the behaviour of return, returning only the value itself instead of key=value as usual.
nice! Another tip I didn't know.
I am still trying to find a way to get the search scheduled time start, but I didn't find it yet. Does scheduled search have a special field containing their scheduled time?
Thank you very much martin for the help. I finally found the rest of the solution from here, use | addinfo, and infomintime to retrieve the starting time of the search. So the final answer is (if you want earliest to start 20 hours before the scheduled time of the search eg):
index=internal earliest=[ search index=internal | head 1| addinfo | eval test=infomintime-20*3600 | return $test]
simplified, optimized, cleaned version:
index=internal earliest=[ stats count | addinfo | eval test=relativetime(infomintime, "-20h") | return $test]