topic Re: File ingested generates another file with the same data which results in data duplication in Getting Data In

File ingested generates another file with the same data which results in data duplication

ejmin — Mon, 20 Jul 2020 19:13:48 GMT

Hi Splunk Experts I have this kind of problem which confuses me. The file being ingested generates another file which has a different filename format but contains the same data. Please see the examples of the data being generated below. (I dont want to use dedup function as well as the scheduled query with piping delete function because it doesnt resolve the root cause problem)

/opt/splunk/CompanyX/DAILY/JUL-20_GPW_DAILY_2020_AS_OF_07212020.txt (original)

/opt/splunk/CompanyX/DAILY/.JUL-20_GPW_DAILY_2020_AS_OF_07212020.txt.tokUbm (generated)

/opt/splunk/CompanyX/DAILY/JUL-19_GPW_DAILY_2020_AS_OF_07202020.txt (original)

/opt/splunk/CompanyX/DAILY/.JUL-19_GPW_DAILY_2020_AS_OF_07202020.txt.MjSIIF(generated)

/opt/splunk/CompanyX/DAILY/JUL-18_GPW_DAILY_2020_AS_OF_07192020.txt (original)

/opt/splunk/CompanyX/DAILY/.JUL-18_GPW_DAILY_2020_AS_OF_07192020.txt.nO9Y5C(generated)

The extraction happens on midnight and goes to a certain directory in which the script replicates into the splunk indexer instance.

My configuration on inputs.conf:

[monitor:///opt/splunk/CompanyX/DAILY/*]

disabled = false

index=gpw_daily

sourcetype=gpw_csv

crcSalt=<SOURCE>

The configuration is working properly for the past year and this incident only happens this past week. So if anyone has encountered this problem please help me to resolve it. Thanks

Re: File ingested generates another file with the same data which results in data duplication

richgalloway — Mon, 20 Jul 2020 21:00:50 GMT

What changed in the past week?

Consider modifying the inputs.conf file to reduce duplication. Either this

[monitor:///opt/splunk/CompanyX/DAILY/*.txt] disabled = false index=gpw_daily sourcetype=gpw_csv crcSalt=<SOURCE>

or this

[monitor:///opt/splunk/CompanyX/DAILY/*] disabled = false index=gpw_daily sourcetype=gpw_csv crcSalt=<SOURCE> whitelist = *.txt

Re: File ingested generates another file with the same data which results in data duplication

ejmin — Tue, 21 Jul 2020 15:55:11 GMT

Hi @richgalloway ,

Thanks for the reply, The duplication happened this past week. It starts generating some .txt.(random) letters but for this past year seems fine. I wonder how it happen because when I searched in the _internal and found its _indextime the ingestion happens within a second interval and produces different bytes according to logs.

But anyway if I dont find the root cause for this problem Ill consider to use your whitelist key-value settings. Thanks Ill consider it as a temporary solution but not the root cause solution.