Getting Data In

Debug timestamp issue for data coming from UDP port in Cloud Splunk.

Explorer

I have an index whose data is being fetched from UDP port. Index is experiencing latency [lag in events] and we suspect timestamp issues with index. How can this be debugged further.

0 Karma

SplunkTrust
SplunkTrust

So all events are showing timestamps a certain amount of hours behind/ahead?

0 Karma

Explorer

there is lag in data indexing [behind].

0 Karma

SplunkTrust
SplunkTrust

Try running something like this and share result.

index=yourindex sourcetype=yoursourcetype | eval lag=abs(_time-_indextime) | stats avg(lag) max(lag) min(lag)

If there is an issue in timestamp parsing, especially the Timezone, then all three columns should be very close in value.

Is your data in json format? (with values in double quotes)?

0 Karma

Explorer

yes Json

avg(lag) : 53.383068783068786
max(lag): 143
min(lag):0

0 Karma

SplunkTrust
SplunkTrust

so per above stats, the lags are less that 3 mins, which are acceptable to many. Do you expect those to be even lesser?

Per above stats also, there doesn't seem to be timezone issue. I would still recommend using following for your props.conf

[yourSourceTypeHere]
KV_MODE = none
TRUNCATE = 0
SHOULD_LINEMERGE = false
TIME_PREFIX = \"ts\"\:\"
MAX_TIMESTAMP_LOOKAHEAD = 28
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%7N%Z
MAX_EVENTS = 1
0 Karma

Explorer

If I check for 24hrs, lag has reached to even 30minutes as well.

avg(lag) :872.6742391843478
max(lag) :1815.000000
min(lag) : 0

Moreover, If its not a timeZone issue, what else could be the reason, is it timestamp parsing issue only?

0 Karma

SplunkTrust
SplunkTrust

It could be. How about you use the props.conf which I provided, which should parse the timestamp correctly (based on our sample data). Monitor for few hours and see if the lag has reduced or not.

Also check if you see any blocked queues on your indexer/heavy forwarder which might be adding delay in indexing.
https://answers.splunk.com/answers/168882/how-to-troubleshoot-blocked-queues-that-are-preven.html

0 Karma

Explorer

Doesn't looks to be a case of blocked queues. We'll try altering props.cfg as you suggested.

Thanks

0 Karma

SplunkTrust
SplunkTrust

please show us some sample data and your props.conf for parsing timestamp

0 Karma

Explorer

Data like :
hostname: NB-9-126-2

level: INFO

monitoring: WrkSet: 73M
PeakWrkSet: 74M
PrivMemSize: 80M
VirtMemSize: 487M
HndleCnt: 649
ThrdCnt: 30

msg: null

ts: 2018-07-18T14:27:25.1923380Z

props.cfg-
KVMODE = none
TRUNCATE = 0
SHOULD
LINEMERGE = false
TIMEPREFIX = "ts":"
MAX
TIMESTAMPLOOKAHEAD = 2048
MAX
EVENTS = 1

0 Karma

SplunkTrust
SplunkTrust

any clue in splunkd.log?
index = _internal sourcetype=splunkd ... other text like your sourcetype or udp port number

0 Karma

Explorer

I am getting some Date Parse warnings.

WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Wed Jul 18 07:00:14 2018)

0 Karma