Splunk Search

How to search when an event was indexed?

a212830
Champion

Hi,

Is there a way to tell when an event is actually indexed? I have a customer who is saying events are showing up with the proper timestamps, but 2 days late. So, I need a way to show when the event was indexed on the system.

Tags (2)
1 Solution

gfuente
Motivator

Hello

You can use the internal field _indextime to get what you need

To make it simplier you can do this:

your base search... | eval _time = _indextime | ...

Regards

View solution in original post

woodcock
Esteemed Legend

Give Meta Woot! a try:
https://splunkbase.splunk.com/app/2949/

We talk about this kind of thing in our talk from .conf18:
https://conf.splunk.com/files/2018/recordings/10-must-have-apps-fn1072.mp4

0 Karma

jtrucks
Splunk Employee
Splunk Employee

Do a chart showing both values to see where they might be off or if there is some pattern. Start with something like this:

* | timechart span=1s values(_indextime) AS indextime, values(_time) AS time

Set the Chart Overlay to be _indextime and Yes show it as an axis to see how they graph out. The table itself is telling as it will show you that there will be slight discrepancies in most data by fractions of a second. However, in your case, it may help you see a pattern.

You could further refine the visual display by using convert() to change the time displays or evals to find the differences between the two values and then chart that. For example, do the following and show as column or bar chart:

* | eval timespan=_indextime-_time | timechart span=1s values(timespan) AS timespan

martin_mueller
SplunkTrust
SplunkTrust

Since I keep pointing people back at this comment all the time, might as well drop in a new-and-improved version:

| tstats count min(_indextime) as min, avg(_indextime) as avg, max(_indextime) as max where index=foo by _time span=1s 
| foreach min avg max 
    [ eval <<FIELD>> = <<FIELD>> - _time] 
| bin span=30m _time as tmp
| eventstats sum(count) as sum by tmp
| eval avg = count * avg / sum
| timechart span=30m min(min) as min sum(avg) as avg max(max) as max

This will not only get you an upper bound for indexing delay, but also a lower bound "events from the future" and a statistically accurate average 🙂

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

To quickly get a general overview of your indexing delay, consider something tstatsy like this:

| tstats max(_indextime) as max where index=foo by host _time span=1s
| eval delta = max - _time | timechart max(delta) by host

gfuente
Motivator

Hello

You can use the internal field _indextime to get what you need

To make it simplier you can do this:

your base search... | eval _time = _indextime | ...

Regards

View solution in original post

a212830
Champion

Thanks. How would I test if the index time is behind the timestamp in the logfile?

0 Karma

Arun_N_007
Communicator

Just Comapare both times and get the difference.

..|eval diff=_indextime-_time|table _indextime,_time,diff

.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!