Difference between lastTime and recentTime in metadata output

[southeringtonp]

Can someone explain the distinction between the lastTime and recentTime fields in the output of the | metadata command?

[ahjmcaleer]

A lot of Splunk articles say that recentTime and localTime will be the same, but that's not true if your devices don't all store data in UTC time.

In our experience, recentTime is relative to the local time of whoever is conducting the search, while lastTime is the latest timestamp reported by the device and stored inside an index.

If you have devices in different timezones (in other words, you don't use GMT/UTC), you need to be careful about the different commands. Example: If you want an alert on devices that haven't reported to an index in the last 1800 seconds, we use recentTime so that everything is relative to the local time:
| metadata index=indexname type=hosts | eval age=now()-recentTime | search age>1800

If you use lastTime, your ages will be all over the place because devices are reporting from different timezones. A negative age means the device is in a timezone ahead of yours, so it thinks the device is in the future.

If you want to see for yourself, try | metadata index=indexname type=hosts | eval age=now()-recentTime and then try | metadata index=indexname type=hosts | eval age=now()-lastTime and see the difference in ages.

Summary
recentTime: Timezone of the search head/indexer
lastTime: Last timestamp seen in the data (potentially a different timezone)

[sloshburch]

Never thought about it that way.

The way I've been thinking about it going forward was that one represented the time of the last seen event (_time) while the other was the time the last event was indexed (_indextime). The docs page has since been updated nicely: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Metadata

The lastTime field is the timestamp for the last time that the indexer saw an event from this host.
The recentTime field is the indextime for the most recent time that the index saw an event from this

[sloshburch]

Let me know if this is better suited as a separate question:

Is it a correct interpretation to say that if using metadata like this:

( index=* OR index=_* ) type=hosts

then the recentTime is the timestamp for the last event from each of the hosts while the lastTime is the last thing we "heard" from that host's forwarder?

Said another way, if a forwarder does not see anything new written to a log file that it monitors, the recentTime will start to get older. If that same forwarder is forwarding it's _internal|_audit data then the lastTime may be more recent due to normal forwarder/indexer communication.

Is that correct? I'm asking because I realize this could be an effective way to monitor for forwarders that stopped communicating with indexers.

[ziegfried]

The question has been answered before: http://answers.splunk.com/questions/5626/what-is-the-difference-between-lasttime-and-recenttime-in-a...

It seems like recentTime is (possibly extracted) timestamp of the last event that has gotten into the index and lastTime is the latest timestamp found in the index - max(_time).

So none of the values would represent max(_indextime) as I understood.

[Lowell]

After looking closer at the values, I think you are correct about the max(_indextime) thing. I've updated my answer accordingly. Thanks for pointing this out.

[Lowell · Accepted solution]

firstTime and lastTime show you the bounds of your timestamps for the entry in question. Think of this like | stats min(_time) as firstTime, max(_time) as lastTime.

The recentTime is the last timestamp that splunk received for the given entry in question. (This would be something like | sort -_indextime | head 1 | rename _time as recentTime.)

Keep in mind that the metadata command is really just pulling in saved statistics about your host/source/sourcetype that are stored within an individual index. (You can see them in the *.data files within the index folder and under individual bucket folders.) Keep in mind that the search examples above are really only meant to give you a comparable idea of what's going on, but they are probably over simplified.

---

So, unless you have events coming in out of order, then lastTime and recentTime will probably contain the same value.