I am ingesting data at 6AM, 2PM, 7PM, 10PM (CST)
Is there anyway I could have my query check the time and set earliest to the time that it past last?
It is 8AM CST time, the search would set earliest to 6AM
It is 9PM CST time, the search would set earliest to 7PM
I'm sure it's possible, but the easier way to do it would be to:
dedupto ensure your results only contain the latest data.
I thought about it a little more and something like this should work. It is certainly not perfect though - you would want to check month and year crossovers also to guarantee accuracy. Change log_level to
INFO if you're environment is perfect and doesn't have errors.
index=_internal log_level=ERROR | eval curr_date_hour = strftime(now(), "%H"), curr_date_mday = strftime(now(), "%d") | eval search_mday = case(curr_date_hour <= 6, curr_date_mday - 1, curr_date_hour > 6, curr_date_mday) | eval search_hour = case(curr_date_hour < 6, 22, curr_date_hour > 22, 22, curr_date_hour > 19, 19, curr_date_hour > 14, 14, curr_date_hour > 6, 6) | where (search_hour != 22 AND search_mday = date_mday AND date_hour >= search_hour) OR (search_hour = 22 AND search_mday = date_mday AND date_hour >= search_hour) OR (search_hour = 22 AND search_mday + 1 = date_mday)