Splunk data forwarding and indexing data drop issue

[manikandankasi]

Hi,

I'm facing issue with data forwarding to splunk. i'm not sure where data being dropped and its happening randomly.
Details:
I have text (key-value pair) file with 6.5 million lines(events) with same timestamp (_time) configured.
but while ingesting file to splunk via Heavy forwarder, it automatically incrementing _time +1 sec for every 100k or 200k events randomly.
Observation:
if the _time +1 sec increment happens for every 100k events, then no issues data completely ingest to splunk.
if some times _time +1 sec increment happens for 200+k events, we are observing data drop, only 4 to 4.5 million events got ingested out of 6.5 million events.

splunk log giving this warning:
WARN DateParserVerbose - The same timestamp has been used for 500K consecutive times. If more than 200K events have the same timestamp, not all events may be retrieveable

Splunk Environment details:
Splunk Version: 7.2.6
OS: AWS Linux Machine

Could you please advice what is root cause of this issue and remedy for same.

Thanks In Advance !!!.
Mani

[codebuilder]

Your sourcetype is not properly extracting the timestamp, OR...your log entries do not include milliseconds.
In either case, Splunk cannot determine unique events, and generates the messages you described.

[harsmarvania57]

I'll suggest here to fix timestamp at source (which is generating this log) so that it will be unique for every event (For example include milliseconds in your timestamp), it will be helpful for few scenario while searching those data.