Getting Data In

Debug timestamp issue for data coming from UDP port in Cloud Splunk.

JuhiSaxena
Explorer

I have an index whose data is being fetched from UDP port. Index is experiencing latency [lag in events] and we suspect timestamp issues with index. How can this be debugged further.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

So all events are showing timestamps a certain amount of hours behind/ahead?

0 Karma

JuhiSaxena
Explorer

there is lag in data indexing [behind].

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try running something like this and share result.

index=yourindex sourcetype=yoursourcetype | eval lag=abs(_time-_indextime) | stats avg(lag) max(lag) min(lag)

If there is an issue in timestamp parsing, especially the Timezone, then all three columns should be very close in value.

Is your data in json format? (with values in double quotes)?

0 Karma

JuhiSaxena
Explorer

yes Json

avg(lag) : 53.383068783068786
max(lag): 143
min(lag):0

0 Karma

somesoni2
SplunkTrust
SplunkTrust

so per above stats, the lags are less that 3 mins, which are acceptable to many. Do you expect those to be even lesser?

Per above stats also, there doesn't seem to be timezone issue. I would still recommend using following for your props.conf

[yourSourceTypeHere]
KV_MODE = none
TRUNCATE = 0
SHOULD_LINEMERGE = false
TIME_PREFIX = \"ts\"\:\"
MAX_TIMESTAMP_LOOKAHEAD = 28
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%7N%Z
MAX_EVENTS = 1
0 Karma

JuhiSaxena
Explorer

If I check for 24hrs, lag has reached to even 30minutes as well.

avg(lag) :872.6742391843478
max(lag) :1815.000000
min(lag) : 0

Moreover, If its not a timeZone issue, what else could be the reason, is it timestamp parsing issue only?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

It could be. How about you use the props.conf which I provided, which should parse the timestamp correctly (based on our sample data). Monitor for few hours and see if the lag has reduced or not.

Also check if you see any blocked queues on your indexer/heavy forwarder which might be adding delay in indexing.
https://answers.splunk.com/answers/168882/how-to-troubleshoot-blocked-queues-that-are-preven.html

0 Karma

JuhiSaxena
Explorer

Doesn't looks to be a case of blocked queues. We'll try altering props.cfg as you suggested.

Thanks

0 Karma

adonio
Ultra Champion

please show us some sample data and your props.conf for parsing timestamp

0 Karma

JuhiSaxena
Explorer

Data like :
hostname: NB-9-126-2

level: INFO

monitoring: WrkSet: 73M
PeakWrkSet: 74M
PrivMemSize: 80M
VirtMemSize: 487M
HndleCnt: 649
ThrdCnt: 30

msg: null

ts: 2018-07-18T14:27:25.1923380Z

props.cfg-
KV_MODE = none
TRUNCATE = 0
SHOULD_LINEMERGE = false
TIME_PREFIX = "ts":"
MAX_TIMESTAMP_LOOKAHEAD = 2048
MAX_EVENTS = 1

0 Karma

adonio
Ultra Champion

any clue in splunkd.log?
index = _internal sourcetype=splunkd ... other text like your sourcetype or udp port number

0 Karma

JuhiSaxena
Explorer

I am getting some Date Parse warnings.

WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Wed Jul 18 07:00:14 2018)

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...