How do I find out why _indextime is greater than _time for a lot of my events?

[abhayneilam]

Hi,

I have a lot of events where "indextime" is > than "eventime". It means something went wrong and it might be one of the below reasons:

1. Forwarder was down ( Because of which event time is not equal to index time , hence indextime > eventime )
2. Backlogs
3. Timestamp is not in the correct format

Please help me out in finding out how to get the exact reason of the difference between indextime and the eventime from the above three listed reasons or if there is any.

Basically, how to identify what is the reason of getting the difference between "indextime" and "eventtime"

Please help me with some good examples. Urgently needed. your help would b highly appreciated !!

Cheers,

[matthieu_araman]

I would graph number of incoming log + the delta

if it's all or nothing -> fwd stop ?

if varies with load -> perf pb somewhere

otherwise or if it's random it may be a timestamp parsing pb

Anyway, in your case, I would start by suspecting the timestamp parsing

[abhayneilam]

Thanks for your prompt reply Mat !!

How will you get the number of incoming logs and Delta ? and what does "if it's all or nothing mean" ? How will you check if it varies with load 😞 .
and more importantly how will you check the wrong timestamp populated in the events ?

Please try to get me some practical stuff, commands or anything which could be useful to solve the stuffs.

Cheers,

[richgalloway]

How big is the difference between event time and index time?

[abhayneilam]

difference could be 2 secs , but we are ignoring 2 secs diff ,

We have kept a threshold of > 5-10 mins

Cheers,
..