Getting Data In
Highlighted

Debug timestamp issue for data coming from UDP port in Cloud Splunk.

Explorer

I have an index whose data is being fetched from UDP port. Index is experiencing latency [lag in events] and we suspect timestamp issues with index. How can this be debugged further.

0 Karma
Highlighted

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

SplunkTrust
SplunkTrust

any clue in splunkd.log?
index = _internal sourcetype=splunkd ... other text like your sourcetype or udp port number

0 Karma
Highlighted

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

Explorer

I am getting some Date Parse warnings.

WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Wed Jul 18 07:00:14 2018)

0 Karma
Highlighted

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

SplunkTrust
SplunkTrust

So all events are showing timestamps a certain amount of hours behind/ahead?

0 Karma
Highlighted

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

SplunkTrust
SplunkTrust

please show us some sample data and your props.conf for parsing timestamp

0 Karma
Highlighted

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

Explorer

Data like :
hostname: NB-9-126-2

level: INFO

monitoring: WrkSet: 73M
PeakWrkSet: 74M
PrivMemSize: 80M
VirtMemSize: 487M
HndleCnt: 649
ThrdCnt: 30

msg: null

ts: 2018-07-18T14:27:25.1923380Z

props.cfg-
KVMODE = none
TRUNCATE = 0
SHOULD
LINEMERGE = false
TIMEPREFIX = "ts":"
MAX
TIMESTAMPLOOKAHEAD = 2048
MAX
EVENTS = 1

0 Karma
Highlighted

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

Explorer

there is lag in data indexing [behind].

0 Karma
Highlighted

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

SplunkTrust
SplunkTrust

Try running something like this and share result.

index=yourindex sourcetype=yoursourcetype | eval lag=abs(_time-_indextime) | stats avg(lag) max(lag) min(lag)

If there is an issue in timestamp parsing, especially the Timezone, then all three columns should be very close in value.

Is your data in json format? (with values in double quotes)?

0 Karma
Highlighted

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

Explorer

yes Json

avg(lag) : 53.383068783068786
max(lag): 143
min(lag):0

0 Karma
Highlighted

Re: Debug timestamp issue for data coming from UDP port in Cloud Splunk.

SplunkTrust
SplunkTrust

so per above stats, the lags are less that 3 mins, which are acceptable to many. Do you expect those to be even lesser?

Per above stats also, there doesn't seem to be timezone issue. I would still recommend using following for your props.conf

[yourSourceTypeHere]
KV_MODE = none
TRUNCATE = 0
SHOULD_LINEMERGE = false
TIME_PREFIX = \"ts\"\:\"
MAX_TIMESTAMP_LOOKAHEAD = 28
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%7N%Z
MAX_EVENTS = 1
0 Karma