topic Re: How to search when an event was indexed? in Splunk Search

How to search when an event was indexed?

a212830 — Wed, 29 Apr 2015 11:10:47 GMT

Hi,

Is there a way to tell when an event is actually indexed? I have a customer who is saying events are showing up with the proper timestamps, but 2 days late. So, I need a way to show when the event was indexed on the system.

Re: How to search when an event was indexed?

gfuente — Wed, 29 Apr 2015 11:31:57 GMT

Hello

You can use the internal field _indextime to get what you need

To make it simplier you can do this:

your base search... | eval _time = _indextime | ...

Regards

Re: How to search when an event was indexed?

a212830 — Wed, 13 May 2015 18:25:15 GMT

Thanks. How would I test if the index time is behind the timestamp in the logfile?

Re: How to search when an event was indexed?

jtrucks — Wed, 13 May 2015 20:14:36 GMT

Do a chart showing both values to see where they might be off or if there is some pattern. Start with something like this:

* | timechart span=1s values(_indextime) AS indextime, values(_time) AS time

Set the Chart Overlay to be _indextime and Yes show it as an axis to see how they graph out. The table itself is telling as it will show you that there will be slight discrepancies in most data by fractions of a second. However, in your case, it may help you see a pattern.

You could further refine the visual display by using convert() to change the time displays or evals to find the differences between the two values and then chart that. For example, do the following and show as column or bar chart:

* | eval timespan=_indextime-_time | timechart span=1s values(timespan) AS timespan

Re: How to search when an event was indexed?

martin_mueller — Wed, 13 May 2015 21:06:38 GMT

To quickly get a general overview of your indexing delay, consider something tstatsy like this:

| tstats max(_indextime) as max where index=foo by host _time span=1s
| eval delta = max - _time | timechart max(delta) by host

Re: How to search when an event was indexed?

Arun_N_007 — Mon, 28 Sep 2020 20:00:06 GMT

Just Comapare both times and get the difference.

..|eval diff=_indextime-_time|table _indextime,_time,diff

Re: How to search when an event was indexed?

martin_mueller — Fri, 19 Oct 2018 19:09:59 GMT

Since I keep pointing people back at this comment all the time, might as well drop in a new-and-improved version:

| tstats count min(_indextime) as min, avg(_indextime) as avg, max(_indextime) as max where index=foo by _time span=1s 
| foreach min avg max 
    [ eval <<FIELD>> = <<FIELD>> - _time] 
| bin span=30m _time as tmp
| eventstats sum(count) as sum by tmp
| eval avg = count * avg / sum
| timechart span=30m min(min) as min sum(avg) as avg max(max) as max

This will not only get you an upper bound for indexing delay, but also a lower bound "events from the future" and a statistically accurate average 🙂

Re: How to search when an event was indexed?

woodcock — Fri, 19 Oct 2018 21:37:02 GMT

Give Meta Woot! a try:
https://splunkbase.splunk.com/app/2949/

We talk about this kind of thing in our talk from .conf18:
https://conf.splunk.com/files/2018/recordings/10-must-have-apps-fn1072.mp4