any clue in splunkd.log?
index = _internal sourcetype=splunkd ... other text like your sourcetype or udp port number
I am getting some Date Parse warnings.
WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Wed Jul 18 07:00:14 2018)
Data like :
monitoring: WrkSet: 73M
KVMODE = none
TRUNCATE = 0
SHOULDLINEMERGE = false
TIMEPREFIX = "ts":"
MAXTIMESTAMPLOOKAHEAD = 2048
MAXEVENTS = 1
Try running something like this and share result.
index=yourindex sourcetype=yoursourcetype | eval lag=abs(_time-_indextime) | stats avg(lag) max(lag) min(lag)
If there is an issue in timestamp parsing, especially the Timezone, then all three columns should be very close in value.
Is your data in json format? (with values in double quotes)?
so per above stats, the lags are less that 3 mins, which are acceptable to many. Do you expect those to be even lesser?
Per above stats also, there doesn't seem to be timezone issue. I would still recommend using following for your props.conf
[yourSourceTypeHere] KV_MODE = none TRUNCATE = 0 SHOULD_LINEMERGE = false TIME_PREFIX = \"ts\"\:\" MAX_TIMESTAMP_LOOKAHEAD = 28 TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%7N%Z MAX_EVENTS = 1