Delay in index time and search time data..
There is a delay of 10 hours
index=test_shift "*10987867*" | eval indextime=strftime(_indextime,"%d/%m/%Y %H:%M:%S")
| table _raw _time indextime
_time is 2022-11-15 13:42:31
indextime is 2022-11-15 23:27:33
The environment is in cluster and if one indexer is down also , but the delay shouldnt be there .
kindly suggest me what is the root cause so that i check in my environment
Thanks in advance and your answer will be helpful for me
There are several possible causes.
In general you can divide them into three groups:
1) The source reports wrong time (for example - it has a misconfigured clock)
2) The time is not properly parsed from the message (the format is not properly defined, the timezone is off)
3) There is simply a delay between when the event was "produced" by the source and the time it went through all the ingestion process. It can mean that the data is getting stuck somewhere but for some types of sources it can be perfectly normal (like in cases where the batch load of events is synchronized in a cron-scheduled manner from the source or with some cases of WEF subscriptions or some periodicaly working modular inputs). Not every source can deliver events continuously.
_time is usually derived from the timestamp data found in the event - how have you defined your sourcetype with respect to finding the timestamp?