topic Re: Duplicate data because of file parts in Getting Data In

Duplicate data because of file parts

strive — Tue, 14 May 2013 22:07:59 GMT

Hi,

I took 6 log files. The sum of events from all the log files is 10666.

I added the log files into my forwarder node.

When i checked the index: index=my_raw_index The data showed was 21332. Double of actual count.

When i checked the source, there are 12 sources instead of 6. Some source types are fileparts of actual log file. Its like mylog.log-20130514.filepart

If i run a query: index=my_raw_index | where like(source, "%20130514"), it gives me 10666.

Since the file size is huge, it took sometime for the log files to get copied completely.

I have following settings in inputs.conf file of forwarder node.

[monitor:///home/data/aaa/bbb/*]
disabled = false
sourcetype = bbb_ccc
index = my_raw_index
crcSalt = SOURCE

Note: For crcSalt, angular brackets are there. Here i took it out since nothing was getting displayed.

How to avoid this filepart indexing. What settings should be enabled so that data is not indexed twice.

Thanks

Strive

Re: Duplicate data because of file parts

the_wolverine — Tue, 14 May 2013 22:19:06 GMT

You most likely should be blacklisting the *.filepart files since they are partial files. You can do this by adding the following line to your monitor stanza:

blacklist = \.(filepart)$

Remove "crcSalt = SOURCE".
You'll need to re-index those log files as Splunk has already seen them and will not re-index them unless you do something like clean the index (if that's possible on this index.)

Re: Duplicate data because of file parts

strive — Tue, 14 May 2013 22:27:29 GMT

Thank you for your response. I will check your solution.
The combined size of 6 log files is 4.5 MB.

Should i use [batch] rather than monitor in this scenario. Actually in production environment it will be around 7MB.

Re: Duplicate data because of file parts

the_wolverine — Tue, 14 May 2013 22:36:59 GMT

You should really only use batch for one time read and destruct of your log files. Please refer to the documentation for batch input to confirm if that's what you really want to do.

Re: Duplicate data because of file parts

strive — Tue, 14 May 2013 22:58:58 GMT

The combined size of 6 log files is 4.5 MB.

In production the combined size of log files would be around 8 MB

Re: Duplicate data because of file parts

strive — Wed, 15 May 2013 07:07:49 GMT

I cleaned the index. Added blacklist = .(filepart)$
I did not remove crcSalt=.

Data is not getting indexed.

Re: Duplicate data because of file parts

strive — Wed, 15 May 2013 10:56:34 GMT

Its working. Thank you.

Re: Duplicate data because of file parts

kristian_kolb — Wed, 15 May 2013 10:59:23 GMT

Did you clean out the fishbucket as well? Unless you do so, Splunk will not re-index the files.

That is an index (which can be cleaned) where splunk stores what it has already seen (files, offset-pointers). Beware though that if you clean this, splunk will re-index any file it's been configured to monitor (if they're still there).

Oh, for reasons that you've just experienced, you should not copy huge files over the network directly into a monitored folder. It's better to copy it to a temp folder (on the same file system) and then move it into the monitored folder.

Re: Duplicate data because of file parts

strive — Thu, 23 May 2013 09:32:43 GMT

Thanks a lot kristian. Your suggestion makes sense to copy the files to temp folder first.

Re: Duplicate data because of file parts

BansodeSantosh — Tue, 09 May 2017 05:36:39 GMT

This solution worked...thanks #the_wolverine