Try running something like this and share result.
index=yourindex sourcetype=yoursourcetype | eval lag=abs(_time-_indextime) | stats avg(lag) max(lag) min(lag)
If there is an issue in timestamp parsing, especially the Timezone, then all three columns should be very close in value.
Is your data in json format? (with values in double quotes)?
so per above stats, the lags are less that 3 mins, which are acceptable to many. Do you expect those to be even lesser?
Per above stats also, there doesn't seem to be timezone issue. I would still recommend using following for your props.conf
[yourSourceTypeHere] KV_MODE = none TRUNCATE = 0 SHOULD_LINEMERGE = false TIME_PREFIX = \"ts\"\:\" MAX_TIMESTAMP_LOOKAHEAD = 28 TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%7N%Z MAX_EVENTS = 1
If I check for 24hrs, lag has reached to even 30minutes as well.
min(lag) : 0
Moreover, If its not a timeZone issue, what else could be the reason, is it timestamp parsing issue only?
It could be. How about you use the props.conf which I provided, which should parse the timestamp correctly (based on our sample data). Monitor for few hours and see if the lag has reduced or not.
Also check if you see any blocked queues on your indexer/heavy forwarder which might be adding delay in indexing.
Data like :
monitoring: WrkSet: 73M
KVMODE = none
TRUNCATE = 0
SHOULDLINEMERGE = false
TIMEPREFIX = "ts":"
MAXTIMESTAMPLOOKAHEAD = 2048
MAXEVENTS = 1