Another thing to bear in mind is what the _time (stamp) means. If it is interpreted from the data in the event, then it is the time that the application has chosen to put into the data. For example,...
See more...
Another thing to bear in mind is what the _time (stamp) means. If it is interpreted from the data in the event, then it is the time that the application has chosen to put into the data. For example, with Apache HTTPD logs (and other logs), the timestamp is when the request was received, but the logged event is written when the response was sent back, so is already lagging by whatever the response time of the request was. 12:01:00 - request received by Apache (_time) 12:01:10 - response received by Apache 12:01:11 - event logged by Apache (request time and duration time of 10 seconds) 12:01:14 - event indexed by Splunk (_indextime) As you can see in this example, the difference between _time and _indextime is 14 seconds, but the lag between when the event was written and when it was indexed is only 3 seconds. So, unless the _time value is the time (or as close as possible to the time) that the application wrote the event (so it was available to the forwarders to send to Splunk), calculating the difference between _time and _indextime can represent a number of factors and you need to understand what the values represent to determine whether they are of any value. Having said that, comparing the difference with historic differences may at least give you an insight as to whether there is any degradation/variation, which might be worth investigating.