How to search when an event was indexed?

[a212830]

Hi,

Is there a way to tell when an event is actually indexed? I have a customer who is saying events are showing up with the proper timestamps, but 2 days late. So, I need a way to show when the event was indexed on the system.

[woodcock]

Give Meta Woot! a try:
https://splunkbase.splunk.com/app/2949/

We talk about this kind of thing in our talk from .conf18:
https://conf.splunk.com/files/2018/recordings/10-must-have-apps-fn1072.mp4

[jtrucks]

Do a chart showing both values to see where they might be off or if there is some pattern. Start with something like this:

* | timechart span=1s values(_indextime) AS indextime, values(_time) AS time

Set the Chart Overlay to be _indextime and Yes show it as an axis to see how they graph out. The table itself is telling as it will show you that there will be slight discrepancies in most data by fractions of a second. However, in your case, it may help you see a pattern.

You could further refine the visual display by using convert() to change the time displays or evals to find the differences between the two values and then chart that. For example, do the following and show as column or bar chart:

* | eval timespan=_indextime-_time | timechart span=1s values(timespan) AS timespan

[martin_mueller]

Since I keep pointing people back at this comment all the time, might as well drop in a new-and-improved version:

| tstats count min(_indextime) as min, avg(_indextime) as avg, max(_indextime) as max where index=foo by _time span=1s 
| foreach min avg max 
    [ eval <<FIELD>> = <<FIELD>> - _time] 
| bin span=30m _time as tmp
| eventstats sum(count) as sum by tmp
| eval avg = count * avg / sum
| timechart span=30m min(min) as min sum(avg) as avg max(max) as max

This will not only get you an upper bound for indexing delay, but also a lower bound "events from the future" and a statistically accurate average 🙂

[martin_mueller]

To quickly get a general overview of your indexing delay, consider something tstatsy like this:

| tstats max(_indextime) as max where index=foo by host _time span=1s
| eval delta = max - _time | timechart max(delta) by host

[gfuente · Accepted solution]

Hello

You can use the internal field _indextime to get what you need

To make it simplier you can do this:

your base search... | eval _time = _indextime | ...

Regards

[a212830]

Thanks. How would I test if the index time is behind the timestamp in the logfile?

[Arun_N_007]

Just Comapare both times and get the difference.

..|eval diff=_indextime-_time|table _indextime,_time,diff