I must be missing something. I have a simple search using a time modifier:
My expectation is when I search, the results will be searched and returned only over the last 30 minutes of events. When I use the time picker and set to ALL TIME, it still only returns the last 30 minutes but searches over ALL EVENTS? Is this the correct behavior?
I looked at the job log and it did have the earliest event as the first event in the index (ALL TIME) with the time modifier. Read this a few times, https://docs.splunk.com/Documentation/Splunk/8.0.3/Search/Specifytimemodifiersinyoursearch and didn't see any verbiage to this behavior.
The verbiage you seek is there at that link.
A time range that you specify in the Search bar, or in a saved search, overrides the time range that is selected in the Time Range Picker.
If you specify
earliest=-30m in your query then the time picker should be ignored. If that's not the case then you may have found a bug and should open a support case.
Maybe I'm not explaining it correctly. The events being returned are always within the time modifier on the search.
Try this(for those watching): index=main earliest=-30m for last 4 hours and look at the job inspector for earliestTime, then do the same for ALL Time. The earliestTime is the first event of the index, which tells me splunk is searching all the events and not limiting to a 30 min window based on _time. It takes much longer too, I usually kill it.
That's what I understood and is not expected behavior. Please open a support case.