topic Splunk data forwarding and indexing data drop issue in Getting Data In

Splunk data forwarding and indexing data drop issue

manikandankasi — Fri, 01 Nov 2019 02:33:14 GMT

Hi,

I'm facing issue with data forwarding to splunk. i'm not sure where data being dropped and its happening randomly.
Details:
I have text (key-value pair) file with 6.5 million lines(events) with same timestamp (_time) configured.
but while ingesting file to splunk via Heavy forwarder, it automatically incrementing _time +1 sec for every 100k or 200k events randomly.
Observation:
if the _time +1 sec increment happens for every 100k events, then no issues data completely ingest to splunk.
if some times _time +1 sec increment happens for 200+k events, we are observing data drop, only 4 to 4.5 million events got ingested out of 6.5 million events.

splunk log giving this warning:
WARN DateParserVerbose - The same timestamp has been used for 500K consecutive times. If more than 200K events have the same timestamp, not all events may be retrieveable

Splunk Environment details:
Splunk Version: 7.2.6
OS: AWS Linux Machine

Could you please advice what is root cause of this issue and remedy for same.

Thanks In Advance !!!.
Mani

Re: Splunk data forwarding and indexing data drop issue

harsmarvania57 — Fri, 01 Nov 2019 08:55:31 GMT

I'll suggest here to fix timestamp at source (which is generating this log) so that it will be unique for every event (For example include milliseconds in your timestamp), it will be helpful for few scenario while searching those data.

Re: Splunk data forwarding and indexing data drop issue

codebuilder — Fri, 15 Nov 2019 05:40:15 GMT

Your sourcetype is not properly extracting the timestamp, OR...your log entries do not include milliseconds.
In either case, Splunk cannot determine unique events, and generates the messages you described.