Debug timestamp issue for data coming from UDP port in Cloud Splunk.

[JuhiSaxena]

I have an index whose data is being fetched from UDP port. Index is experiencing latency [lag in events] and we suspect timestamp issues with index. How can this be debugged further.

[somesoni2]

So all events are showing timestamps a certain amount of hours behind/ahead?

[JuhiSaxena]

there is lag in data indexing [behind].

[somesoni2]

Try running something like this and share result.

index=yourindex sourcetype=yoursourcetype | eval lag=abs(_time-_indextime) | stats avg(lag) max(lag) min(lag)

If there is an issue in timestamp parsing, especially the Timezone, then all three columns should be very close in value.

Is your data in json format? (with values in double quotes)?

[JuhiSaxena]

yes Json

avg(lag) : 53.383068783068786
max(lag): 143
min(lag):0

[somesoni2]

so per above stats, the lags are less that 3 mins, which are acceptable to many. Do you expect those to be even lesser?

Per above stats also, there doesn't seem to be timezone issue. I would still recommend using following for your props.conf

[yourSourceTypeHere]
KV_MODE = none
TRUNCATE = 0
SHOULD_LINEMERGE = false
TIME_PREFIX = \"ts\"\:\"
MAX_TIMESTAMP_LOOKAHEAD = 28
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%7N%Z
MAX_EVENTS = 1

[JuhiSaxena]

If I check for 24hrs, lag has reached to even 30minutes as well.

avg(lag) :872.6742391843478
max(lag) :1815.000000
min(lag) : 0

Moreover, If its not a timeZone issue, what else could be the reason, is it timestamp parsing issue only?

[somesoni2]

It could be. How about you use the props.conf which I provided, which should parse the timestamp correctly (based on our sample data). Monitor for few hours and see if the lag has reduced or not.

Also check if you see any blocked queues on your indexer/heavy forwarder which might be adding delay in indexing.
https://answers.splunk.com/answers/168882/how-to-troubleshoot-blocked-queues-that-are-preven.html

[JuhiSaxena]

Doesn't looks to be a case of blocked queues. We'll try altering props.cfg as you suggested.

Thanks

[adonio]

please show us some sample data and your props.conf for parsing timestamp

[JuhiSaxena]

Data like :
hostname: NB-9-126-2

level: INFO

monitoring: WrkSet: 73M
PeakWrkSet: 74M
PrivMemSize: 80M
VirtMemSize: 487M
HndleCnt: 649
ThrdCnt: 30

msg: null

ts: 2018-07-18T14:27:25.1923380Z

props.cfg-
KV_MODE = none
TRUNCATE = 0
SHOULD_LINEMERGE = false
TIME_PREFIX = "ts":"
MAX_TIMESTAMP_LOOKAHEAD = 2048
MAX_EVENTS = 1

[adonio]

any clue in splunkd.log?
index = _internal sourcetype=splunkd ... other text like your sourcetype or udp port number

[JuhiSaxena]

I am getting some Date Parse warnings.

WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Wed Jul 18 07:00:14 2018)