I would like to ask about the timemodifier.
I have a following search including subsearch,
[ search index=hoge indexearliesst=-1d@d indexlatest=@d
earliest(start) as earliest
latest(stop) as latest by field
| eval earliest=substr(earliest,5,2) . "/" . substr(earliest,7,2) . "/" . substr(earliest,1,4) . ":" . substr(earliest,9,2) . ":" . substr(earliest,11,2) . ":" . substr(earliest,13,2)
| search conditionA
| eval latest=substr(latest,5,2) . "/" . substr(latest,7,2) . "/" . substr(latest,1,4) . ":" . substr(latest,9,2) . ":" . substr(latest,11,2) . ":" . substr(latest,13,2)
| fields field earliest latest
| format "(" "(" "" ")" "OR" ")"
My purpose is to search the events that meets the conditionA that were indexed the previous day
and pass the earliest and latest time of each field to the main search.
However, when the number of events should the main search returns are 5000 , it scans more number of events.
field earliest latest
fieldA 1/25/2014 00:00 1/25/2014 01:00 3 records exists
fieldB 1/25/2014 02:00 1/25/2014 02:00 5 recoreds exists
fieldC 1/26/2014 00:00 1/26/2014 01:00
if I expect the subsearch to return (fields="fieldA" earliest="1/25/2014:00:00" latest="1/25/2014:01:00") OR ( field="fieldB" earliest="1/25/2014:02:00" latest="1/25/2014 02:00") , I expect the main search to scan only 8 records, But it seems that it scans the event more than I expect.
Is the timemodifier not working corrctly if you concatenate with OR's?
Right - based on your comments, you're probably looking for the
map command. Consider these two simple searches:
index=_internal (earliest=-30d@d latest=-29d@d) OR (earliest=@d latest=now) | bin _time span=1d | stats count by _time
That's roughly what you're doing now, tell the search to OR two timeranges... and as you observed, this scans the events in between rather than skipping to the past instantaneously. On my PC, this takes about four seconds.
| stats count | eval times="-30d@d|-29d@d,@d|now" | makemv delim="," times | mvexpand times | makemv delim="|" times | eval starttime=mvindex(times,0) | eval endtime=mvindex(times,1) | map search="search index=_internal earliest=$starttime$ latest=$endtime$ | bin _time span=1d | stats count by _time"
This achieves the same thing, but in a different way. The first bit before
map creates two events with fields
endtime, which are used by
map to run two searches based on those timeranges. The result is the same as in the earlier search, but my PC only takes about one second.
This should be applicable to your problem, run the current subsearch first and pass into
map the earliest/latest times as well as the condition.
Not quite sure about a thousand...
map will run a search for each event it's given, so it would run a thousand searches. By default it stops after ten, so running a thousand may or may not break things. Just give it a shot...
While you're studying
map, take a look at
localize - maybe that could simplify your subsearch for building the timeranges.
I have to do some studying with this though. By the way would this work with about thousand set of following conditions( all concatenated with OR)?
(field=hoge1 earliest=xxxx latest=xxxx)
Good, then I don't need to think about that 🙂
Are you basically trying to do this: "Run a subsearch, get a bunch of results, use each result to build a timerange and possibly further filters, and run a search off that"?
Yes. The subsearch returns the set of
earliest time in relative time format
latest time in relative time format
format command is working fine.
It seems that the time modifier is not working as expected in each AND boolean expression. Maybe putting to much time modifier confuses the search...