Hi Splunk Experts I have this kind of problem which confuses me. The file being ingested generates another file which has a different filename format but contains the same data. Please see the examples of the data being generated below. (I dont want to use dedup function as well as the scheduled query with piping delete function because it doesnt resolve the root cause problem)
/opt/splunk/CompanyX/DAILY/JUL-20_GPW_DAILY_2020_AS_OF_07212020.txt (original)
/opt/splunk/CompanyX/DAILY/.JUL-20_GPW_DAILY_2020_AS_OF_07212020.txt.tokUbm (generated)
/opt/splunk/CompanyX/DAILY/JUL-19_GPW_DAILY_2020_AS_OF_07202020.txt (original)
/opt/splunk/CompanyX/DAILY/.JUL-19_GPW_DAILY_2020_AS_OF_07202020.txt.MjSIIF(generated)
/opt/splunk/CompanyX/DAILY/JUL-18_GPW_DAILY_2020_AS_OF_07192020.txt (original)
/opt/splunk/CompanyX/DAILY/.JUL-18_GPW_DAILY_2020_AS_OF_07192020.txt.nO9Y5C(generated)
The extraction happens on midnight and goes to a certain directory in which the script replicates into the splunk indexer instance.
My configuration on inputs.conf:
[monitor:///opt/splunk/CompanyX/DAILY/*]
disabled = false
index=gpw_daily
sourcetype=gpw_csv
crcSalt=<SOURCE>
The configuration is working properly for the past year and this incident only happens this past week. So if anyone has encountered this problem please help me to resolve it. Thanks
Hi @richgalloway ,
Thanks for the reply, The duplication happened this past week. It starts generating some .txt.(random) letters but for this past year seems fine. I wonder how it happen because when I searched in the _internal and found its _indextime the ingestion happens within a second interval and produces different bytes according to logs.
But anyway if I dont find the root cause for this problem Ill consider to use your whitelist key-value settings. Thanks Ill consider it as a temporary solution but not the root cause solution.
What changed in the past week?
Consider modifying the inputs.conf file to reduce duplication. Either this
[monitor:///opt/splunk/CompanyX/DAILY/*.txt]
disabled = false
index=gpw_daily
sourcetype=gpw_csv
crcSalt=<SOURCE>
or this
[monitor:///opt/splunk/CompanyX/DAILY/*]
disabled = false
index=gpw_daily
sourcetype=gpw_csv
crcSalt=<SOURCE>
whitelist = *.txt
Hi @richgalloway ,
Thanks for the reply, The duplication happened this past week. It starts generating some .txt.(random) letters but for this past year seems fine. I wonder how it happen because when I searched in the _internal and found its _indextime the ingestion happens within a second interval and produces different bytes according to logs.
But anyway if I dont find the root cause for this problem Ill consider to use your whitelist key-value settings. Thanks Ill consider it as a temporary solution but not the root cause solution.