File ingested generates another file with the same data which results in data duplication

[ejmin]

Hi Splunk Experts I have this kind of problem which confuses me. The file being ingested generates another file which has a different filename format but contains the same data. Please see the examples of the data being generated below. (I dont want to use dedup function as well as the scheduled query with piping delete function because it doesnt resolve the root cause problem)

/opt/splunk/CompanyX/DAILY/JUL-20_GPW_DAILY_2020_AS_OF_07212020.txt (original)

/opt/splunk/CompanyX/DAILY/.JUL-20_GPW_DAILY_2020_AS_OF_07212020.txt.tokUbm (generated)

/opt/splunk/CompanyX/DAILY/JUL-19_GPW_DAILY_2020_AS_OF_07202020.txt (original)

/opt/splunk/CompanyX/DAILY/.JUL-19_GPW_DAILY_2020_AS_OF_07202020.txt.MjSIIF(generated)

/opt/splunk/CompanyX/DAILY/JUL-18_GPW_DAILY_2020_AS_OF_07192020.txt (original)

/opt/splunk/CompanyX/DAILY/.JUL-18_GPW_DAILY_2020_AS_OF_07192020.txt.nO9Y5C(generated)

The extraction happens on midnight and goes to a certain directory in which the script replicates into the splunk indexer instance. 

My configuration on  inputs.conf:

[monitor:///opt/splunk/CompanyX/DAILY/*]

disabled = false

index=gpw_daily

sourcetype=gpw_csv

crcSalt=<SOURCE>

The configuration is working properly for the past year and this incident only happens this past week.  So if anyone has encountered this problem please help me to resolve it. Thanks

[richgalloway]

What changed in the past week?

Consider modifying the inputs.conf file to reduce duplication.  Either this

[monitor:///opt/splunk/CompanyX/DAILY/*.txt]
disabled = false
index=gpw_daily
sourcetype=gpw_csv
crcSalt=<SOURCE>

or this

[monitor:///opt/splunk/CompanyX/DAILY/*]
disabled = false
index=gpw_daily
sourcetype=gpw_csv
crcSalt=<SOURCE>
whitelist = *.txt

[ejmin · Accepted solution]

Hi @richgalloway ,

Thanks for the reply, The duplication happened this past week. It starts generating some .txt.(random) letters but for this past year seems fine. I wonder how it happen because when I searched in the _internal and found its _indextime the ingestion happens within a second interval and produces different bytes according to logs.

But anyway if I dont find the root cause for this problem Ill consider to use your whitelist key-value settings. Thanks Ill consider it as a temporary solution but not the root cause solution.