These could be real time searches.
I ran a search like "index=*" for 30 seconds realtime, and the apiStartTime was displayed as Zero_time
search totalruntime time apiStartTime apiEndTime searchtype user
search index=* 2018-03-20 10:28:09.913 ZEROTIME ZEROTIME ad hoc testuser01
search index=* 2018-03-20 10:28:13.560 ZEROTIME ZEROTIME ad hoc testuser01
The audit log captures the time range of the search. As a Splunk user, you specify the time range by using the pull-down menu (or by using the
latest keywords). When Splunk processes the search, it calculates the actual time that should be searched.
apiStartTime represents the earliest time, and
apiEndTime represents the latest time.
EDIT - in my original answer, I said
apiStartTime='ZERO_TIME', apiEndTime='ZERO_TIME' means that the search ran over All Time. It makes sense that this would be an excessively expensive search.
but this appears not to be the case.
Sorry but, indeed, it seems that your original answer is wrong.
A simpler search, without apiStartTime='ZEROTIME' apiEndTime='ZEROTIME', returns a bunch of other records, including the very same query, with the exact time range selected by the user. And this query occured just microseconds before the one with ZERO_TIME. So it must be something splunk does, but because it happens all the time it can't mean that it's the "All time" time range that was used.
So I have to remove the point. I will add this in a splunk ticket I opened to resolve cold storage searches that take our system down.
This gets weirder and weirder, according to my last search, and if apiStartTime='ZEROTIME', apiEndTime='ZEROTIME' means "All time", even I ran "All time" queries. This is starting to sound more and more like a bug.
Thank you very much lguinn.
The weird thing is that I disabled the "All time" from the GUI. And the user, from being the previous Splunk admin knows very well not to run "All time" queries. And he confirmed that when asked. So how else could this happen?
Is there any way I can get the exact query that was executed, ie, with the time range specified by the user?