How can manual data uploads with overlapping log files include only unique data? The goal is to avoid uploading duplicate data. For example, if log file #1 with data from Jan 1 - Jan 31 is uploaded and later log file #2 with data from Jan 15 - Feb 15 is uploaded, how can the duplicate data (Jan 15-31) in file #2 automatically not be uploaded? Note that we have a unique requirement that a log file is air gapped from any network, which does not allow us to use a Splunk forwarder.
Splunk doesn't deduplicate on upload. Different log files are assumed to have different data even if the times overlap.
Even if you were allowed to use one, the forwarder wouldn't help.
--- If this reply helps you, Karma would be appreciated.
