topic Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk. in Getting Data In

Debug timestamp issue for data coming from UDP port in Cloud Splunk.

JuhiSaxena — Wed, 18 Jul 2018 11:50:35 GMT

I have an index whose data is being fetched from UDP port. Index is experiencing latency [lag in events] and we suspect timestamp issues with index. How can this be debugged further.

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

adonio — Wed, 18 Jul 2018 13:38:12 GMT

any clue in splunkd.log?
index = _internal sourcetype=splunkd ... other text like your sourcetype or udp port number

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

somesoni2 — Wed, 18 Jul 2018 14:02:37 GMT

So all events are showing timestamps a certain amount of hours behind/ahead?

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

JuhiSaxena — Wed, 18 Jul 2018 14:14:05 GMT

I am getting some Date Parse warnings.

WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Wed Jul 18 07:00:14 2018)

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

adonio — Wed, 18 Jul 2018 14:22:42 GMT

please show us some sample data and your props.conf for parsing timestamp

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

JuhiSaxena — Tue, 29 Sep 2020 20:30:47 GMT

Data like :
hostname: NB-9-126-2

level: INFO

monitoring: WrkSet: 73M
PeakWrkSet: 74M
PrivMemSize: 80M
VirtMemSize: 487M
HndleCnt: 649
ThrdCnt: 30

msg: null

ts: 2018-07-18T14:27:25.1923380Z

props.cfg-
KV_MODE = none
TRUNCATE = 0
SHOULD_LINEMERGE = false
TIME_PREFIX = "ts":"
MAX_TIMESTAMP_LOOKAHEAD = 2048
MAX_EVENTS = 1

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

JuhiSaxena — Wed, 18 Jul 2018 14:34:31 GMT

there is lag in data indexing [behind].

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

somesoni2 — Wed, 18 Jul 2018 14:56:28 GMT

Try running something like this and share result.

index=yourindex sourcetype=yoursourcetype | eval lag=abs(_time-_indextime) | stats avg(lag) max(lag) min(lag)

If there is an issue in timestamp parsing, especially the Timezone, then all three columns should be very close in value.

Is your data in json format? (with values in double quotes)?

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

JuhiSaxena — Wed, 18 Jul 2018 15:05:05 GMT

yes Json

avg(lag) : 53.383068783068786
max(lag): 143
min(lag):0

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

somesoni2 — Wed, 18 Jul 2018 15:11:55 GMT

so per above stats, the lags are less that 3 mins, which are acceptable to many. Do you expect those to be even lesser?

Per above stats also, there doesn't seem to be timezone issue. I would still recommend using following for your props.conf

[yourSourceTypeHere]
KV_MODE = none
TRUNCATE = 0
SHOULD_LINEMERGE = false
TIME_PREFIX = \"ts\"\:\"
MAX_TIMESTAMP_LOOKAHEAD = 28
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%7N%Z
MAX_EVENTS = 1

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

JuhiSaxena — Wed, 18 Jul 2018 15:21:14 GMT

If I check for 24hrs, lag has reached to even 30minutes as well.

avg(lag) :872.6742391843478
max(lag) :1815.000000
min(lag) : 0

Moreover, If its not a timeZone issue, what else could be the reason, is it timestamp parsing issue only?

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

somesoni2 — Wed, 18 Jul 2018 15:43:08 GMT

It could be. How about you use the props.conf which I provided, which should parse the timestamp correctly (based on our sample data). Monitor for few hours and see if the lag has reduced or not.

Also check if you see any blocked queues on your indexer/heavy forwarder which might be adding delay in indexing.
https://answers.splunk.com/answers/168882/how-to-troubleshoot-blocked-queues-that-are-preven.html

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

JuhiSaxena — Wed, 18 Jul 2018 15:53:40 GMT

Doesn't looks to be a case of blocked queues. We'll try altering props.cfg as you suggested.

Thanks