topic Re: How to set up a stanza batch to index same file? in Getting Data In

How to set up a stanza batch to index same file?

arthurf — Mon, 01 Jul 2019 12:39:40 GMT

Hello,

I set up my batch stanza to delete the file once indexed, but sometime, the file can't be due to permission error and the same file is indexed multiple time.

I used both of the following options:

initCrcLength = 2000
crcSalt =

But the same file is still indexed again.
How can I set up the stanza not to read the same file?
Is that possible with batch stanza which deletes the files? (move_policy = sinkhole)

Re: How to set up a stanza batch to index same file?

tiagofbmm — Wed, 30 Sep 2020 01:08:31 GMT

The move_policy MUST be sinkhole for that to work as you want.

move_policy = sinkhole
* This setting is required. You must include "move_policy = sinkhole"
when you define batch inputs.

Re: How to set up a stanza batch to index same file?

arthurf — Mon, 01 Jul 2019 13:20:20 GMT

The move_policy is set.
Here my actual configuration :

[batch://FOLDER/Pattern]
sourcetype = fluxeditique:cabestan:dtr:in:planapurement3
index=cabestan
initCrcLength = 2000
move_policy = sinkhole

I replaced "crcSalt = " with "initCrcLength = 2000" but the result is the same.

Re: How to set up a stanza batch to index same file?

tiagofbmm — Mon, 01 Jul 2019 14:05:40 GMT

CRC settings belong to the monitor::// stanzas, not the batch ones.

It seems you need to take care of the deleting permissions of the files for Splunk user to be able to delete them.

Re: How to set up a stanza batch to index same file?

arthurf — Mon, 01 Jul 2019 15:16:39 GMT

Ok, so if get you correctly, it's not possible to prevent the indexation of the same file with batch stanza ?
And we need to ensure that the same log file is not generated twice, right ?

Re: How to set up a stanza batch to index same file?

tiagofbmm — Mon, 01 Jul 2019 15:21:02 GMT

Yes, with batch you don't get that possibility

Re: How to set up a stanza batch to index same file?

arthurf — Tue, 02 Jul 2019 07:30:49 GMT

Thanks for the confirmation.

Have a good day.