I'm trying to figure out which search will most accurately tell me when events with future timestamps are being detected.
Somebody on the team built this search:
| tstats min(_time) as earliest_time, max(_time) as latest_time by index
| eval daysOfLogs=round((latest_time - earliest_time)/60/60/24, 2)
| eval eventsInFuture=if(latest_time > now(), "yes", "no")
| eval tnow = now()
| convert ctime(*time)
| lookup index_to_env index
| convert ctime(tnow)
That search doesn't show any sourcetypes with future data.
This search, on the other hand, shows that multiple sourcetypes are showing future timestamps:
| metadata type=sourcetypes index=* index!=_* | eval now=now() | eval futuretime=lastTime-now | where futuretime>0
Based on what I've seen searching on the raw events with a "latest=+20d@d", the tstats command is the one that isn't seeing the future events...
Any idea what is causing this behavior?
You can very easily add these to your tstats searches. For example..
| tstats min(_time) as earliest_time, max(_time) as latest_time WHERE earliest=-30d@d latest=+1y@y by index sourcetype | eval daysOfLogs=round((latest_time - earliest_time)/60/60/24, 2) | eval eventsInFuture=if(latest_time > now(), "yes", "no") | eval tnow = now() | convert ctime(*time) | convert ctime(tnow)
You can very much add earliest and latest time boundaries in tstats with the WHERE clause, and you can update BY clause to include any of the indexed fields ( _time, source, sourcetype, host.) Additionally, tstats doesn't show the indexed time, it shows the parsed event time as searched for with min and max and the default behavior is to use the timerange picker's earliest and latest, which is why you wont see future events by default.
Try the above search in your environment.
metadata and dbinspect return a timestamp of the latest event:
Edit: as @esix_splunk mentioned in the post below, this statement is wrong: tstats shows last _indextime, so it is not suitable for your purpose