Splunk Search

How to search when an event was indexed?

a212830
Champion

Hi,

Is there a way to tell when an event is actually indexed? I have a customer who is saying events are showing up with the proper timestamps, but 2 days late. So, I need a way to show when the event was indexed on the system.

Tags (2)
1 Solution

gfuente
Motivator

Hello

You can use the internal field _indextime to get what you need

To make it simplier you can do this:

your base search... | eval _time = _indextime | ...

Regards

View solution in original post

woodcock
Esteemed Legend

Give Meta Woot! a try:
https://splunkbase.splunk.com/app/2949/

We talk about this kind of thing in our talk from .conf18:
https://conf.splunk.com/files/2018/recordings/10-must-have-apps-fn1072.mp4

0 Karma

jtrucks
Splunk Employee
Splunk Employee

Do a chart showing both values to see where they might be off or if there is some pattern. Start with something like this:

* | timechart span=1s values(_indextime) AS indextime, values(_time) AS time

Set the Chart Overlay to be _indextime and Yes show it as an axis to see how they graph out. The table itself is telling as it will show you that there will be slight discrepancies in most data by fractions of a second. However, in your case, it may help you see a pattern.

You could further refine the visual display by using convert() to change the time displays or evals to find the differences between the two values and then chart that. For example, do the following and show as column or bar chart:

* | eval timespan=_indextime-_time | timechart span=1s values(timespan) AS timespan

martin_mueller
SplunkTrust
SplunkTrust

Since I keep pointing people back at this comment all the time, might as well drop in a new-and-improved version:

| tstats count min(_indextime) as min, avg(_indextime) as avg, max(_indextime) as max where index=foo by _time span=1s 
| foreach min avg max 
    [ eval <<FIELD>> = <<FIELD>> - _time] 
| bin span=30m _time as tmp
| eventstats sum(count) as sum by tmp
| eval avg = count * avg / sum
| timechart span=30m min(min) as min sum(avg) as avg max(max) as max

This will not only get you an upper bound for indexing delay, but also a lower bound "events from the future" and a statistically accurate average 🙂

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

To quickly get a general overview of your indexing delay, consider something tstatsy like this:

| tstats max(_indextime) as max where index=foo by host _time span=1s
| eval delta = max - _time | timechart max(delta) by host

gfuente
Motivator

Hello

You can use the internal field _indextime to get what you need

To make it simplier you can do this:

your base search... | eval _time = _indextime | ...

Regards

a212830
Champion

Thanks. How would I test if the index time is behind the timestamp in the logfile?

0 Karma

Arun_N_007
Communicator

Just Comapare both times and get the difference.

..|eval diff=_indextime-_time|table _indextime,_time,diff

Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...