Getting Data In

File ingested generates another file with the same data which results in data duplication

ejmin
Path Finder

Hi Splunk Experts I have this kind of problem which confuses me. The file being ingested generates another file which has a different filename format but contains the same data. Please see the examples of the data being generated below. (I dont want to use dedup function as well as the scheduled query with piping delete function because it doesnt resolve the root cause problem)

/opt/splunk/CompanyX/DAILY/JUL-20_GPW_DAILY_2020_AS_OF_07212020.txt (original)

/opt/splunk/CompanyX/DAILY/.JUL-20_GPW_DAILY_2020_AS_OF_07212020.txt.tokUbm (generated)

/opt/splunk/CompanyX/DAILY/JUL-19_GPW_DAILY_2020_AS_OF_07202020.txt (original)

/opt/splunk/CompanyX/DAILY/.JUL-19_GPW_DAILY_2020_AS_OF_07202020.txt.MjSIIF(generated)

/opt/splunk/CompanyX/DAILY/JUL-18_GPW_DAILY_2020_AS_OF_07192020.txt (original)

/opt/splunk/CompanyX/DAILY/.JUL-18_GPW_DAILY_2020_AS_OF_07192020.txt.nO9Y5C(generated)

 

The extraction happens on midnight and goes to a certain directory in which the script replicates into the splunk indexer instance. 

My configuration on  inputs.conf:

[monitor:///opt/splunk/CompanyX/DAILY/*]

disabled = false

index=gpw_daily

sourcetype=gpw_csv

crcSalt=<SOURCE>

 

The configuration is working properly for the past year and this incident only happens this past week.  So if anyone has encountered this problem please help me to resolve it. Thanks

 

0 Karma
1 Solution

ejmin
Path Finder

Hi @richgalloway ,

Thanks for the reply, The duplication happened this past week. It starts generating some .txt.(random) letters but for this past year seems fine. I wonder how it happen because when I searched in the _internal and found its _indextime the ingestion happens within a second interval and produces different bytes according to logs.

But anyway if I dont find the root cause for this problem Ill consider to use your whitelist key-value settings. Thanks Ill consider it as a temporary solution but not the root cause solution. 

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

What changed in the past week?

Consider modifying the inputs.conf file to reduce duplication.  Either this

[monitor:///opt/splunk/CompanyX/DAILY/*.txt]
disabled = false
index=gpw_daily
sourcetype=gpw_csv
crcSalt=<SOURCE>

or this

[monitor:///opt/splunk/CompanyX/DAILY/*]
disabled = false
index=gpw_daily
sourcetype=gpw_csv
crcSalt=<SOURCE>
whitelist = *.txt

 

---
If this reply helps you, Karma would be appreciated.

ejmin
Path Finder

Hi @richgalloway ,

Thanks for the reply, The duplication happened this past week. It starts generating some .txt.(random) letters but for this past year seems fine. I wonder how it happen because when I searched in the _internal and found its _indextime the ingestion happens within a second interval and produces different bytes according to logs.

But anyway if I dont find the root cause for this problem Ill consider to use your whitelist key-value settings. Thanks Ill consider it as a temporary solution but not the root cause solution. 

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...