https://community.splunk.com/t5/Getting-Data-In/How-can-I-keep-my-data-unique-with-manually-uploading-two/m-p/433455#M75760 <P>How can manual data uploads with overlapping log files include only unique data? The goal is to avoid uploading duplicate data. For example, if log file #1 with data from Jan 1 - Jan 31 is uploaded and later log file #2 with data from Jan 15 - Feb 15 is uploaded, how can the duplicate data (Jan 15-31) in file #2 automatically not be uploaded? Note that we have a unique requirement that a log file is air gapped from any network, which does not allow us to use a Splunk forwarder. </P> Tue, 06 Aug 2019 12:53:00 GMT mkawamura 2019-08-06T12:53:00Z https://community.splunk.com/t5/Getting-Data-In/How-can-I-keep-my-data-unique-with-manually-uploading-two/m-p/433455#M75760 <P>How can manual data uploads with overlapping log files include only unique data? The goal is to avoid uploading duplicate data. For example, if log file #1 with data from Jan 1 - Jan 31 is uploaded and later log file #2 with data from Jan 15 - Feb 15 is uploaded, how can the duplicate data (Jan 15-31) in file #2 automatically not be uploaded? Note that we have a unique requirement that a log file is air gapped from any network, which does not allow us to use a Splunk forwarder. </P> Tue, 06 Aug 2019 12:53:00 GMT https://community.splunk.com/t5/Getting-Data-In/How-can-I-keep-my-data-unique-with-manually-uploading-two/m-p/433455#M75760 mkawamura 2019-08-06T12:53:00Z https://community.splunk.com/t5/Getting-Data-In/How-can-I-keep-my-data-unique-with-manually-uploading-two/m-p/433456#M75761 <P>Splunk doesn't deduplicate on upload. Different log files are assumed to have different data even if the times overlap.<BR /> Even if you were allowed to use one, the forwarder wouldn't help.</P> Tue, 06 Aug 2019 16:41:12 GMT https://community.splunk.com/t5/Getting-Data-In/How-can-I-keep-my-data-unique-with-manually-uploading-two/m-p/433456#M75761 richgalloway 2019-08-06T16:41:12Z