Splunk Search

How to search when an event was indexed?

a212830
Champion

Hi,

Is there a way to tell when an event is actually indexed? I have a customer who is saying events are showing up with the proper timestamps, but 2 days late. So, I need a way to show when the event was indexed on the system.

Tags (2)
1 Solution

gfuente
Motivator

Hello

You can use the internal field _indextime to get what you need

To make it simplier you can do this:

your base search... | eval _time = _indextime | ...

Regards

View solution in original post

woodcock
Esteemed Legend

Give Meta Woot! a try:
https://splunkbase.splunk.com/app/2949/

We talk about this kind of thing in our talk from .conf18:
https://conf.splunk.com/files/2018/recordings/10-must-have-apps-fn1072.mp4

0 Karma

jtrucks
Splunk Employee
Splunk Employee

Do a chart showing both values to see where they might be off or if there is some pattern. Start with something like this:

* | timechart span=1s values(_indextime) AS indextime, values(_time) AS time

Set the Chart Overlay to be _indextime and Yes show it as an axis to see how they graph out. The table itself is telling as it will show you that there will be slight discrepancies in most data by fractions of a second. However, in your case, it may help you see a pattern.

You could further refine the visual display by using convert() to change the time displays or evals to find the differences between the two values and then chart that. For example, do the following and show as column or bar chart:

* | eval timespan=_indextime-_time | timechart span=1s values(timespan) AS timespan
--
Jesse Trucks
Minister of Magic

martin_mueller
SplunkTrust
SplunkTrust

Since I keep pointing people back at this comment all the time, might as well drop in a new-and-improved version:

| tstats count min(_indextime) as min, avg(_indextime) as avg, max(_indextime) as max where index=foo by _time span=1s 
| foreach min avg max 
    [ eval <<FIELD>> = <<FIELD>> - _time] 
| bin span=30m _time as tmp
| eventstats sum(count) as sum by tmp
| eval avg = count * avg / sum
| timechart span=30m min(min) as min sum(avg) as avg max(max) as max

This will not only get you an upper bound for indexing delay, but also a lower bound "events from the future" and a statistically accurate average 🙂

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

To quickly get a general overview of your indexing delay, consider something tstatsy like this:

| tstats max(_indextime) as max where index=foo by host _time span=1s
| eval delta = max - _time | timechart max(delta) by host

gfuente
Motivator

Hello

You can use the internal field _indextime to get what you need

To make it simplier you can do this:

your base search... | eval _time = _indextime | ...

Regards

a212830
Champion

Thanks. How would I test if the index time is behind the timestamp in the logfile?

0 Karma

Arun_N_007
Communicator

Just Comapare both times and get the difference.

..|eval diff=_indextime-_time|table _indextime,_time,diff

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...