topic Re: Blacklist files greater than a certain size from inputs.conf in Splunk Search

Blacklist files greater than a certain size from inputs.conf

edoardo_vicendo — Mon, 25 Jun 2018 14:43:25 GMT

Hi All,

I have to monitor a folder where there are very huge files with file name automatically generated.
Is there some way (instead of write a custom UNIX script that moves only small files to another folder that will be then monitored by the forwarder) to blacklist files that have a size greater than (suppose) 10 MB?

Any other suggestion with Splunk stanza attributes is appreciated.

Thanks a lot,
Edoardo

Re: Blacklist files greater than a certain size from inputs.conf

adonio — Mon, 25 Jun 2018 23:44:09 GMT

hello there,
if the files have some naming convention you can follow, you can apply rules based on their name.

Re: Blacklist files greater than a certain size from inputs.conf

HiroshiSatoh — Tue, 26 Jun 2018 01:10:24 GMT

It can not be executed because there is no size limit parameter in "monitor".
If it is "fschange", it may be restricted by "endEventMaxSize". However, it captures the entire file, not differential import.

Re: Blacklist files greater than a certain size from inputs.conf

edoardo_vicendo — Tue, 26 Jun 2018 11:57:14 GMT

Hi Adonio,

Unfortunately there is no logic in the file name.

Thanks a lot,
Edoardo

Re: Blacklist files greater than a certain size from inputs.conf

adonio — Tue, 26 Jun 2018 12:16:34 GMT

@HiroshiSatoh has the right answer imho

Re: Blacklist files greater than a certain size from inputs.conf

edoardo_vicendo — Tue, 26 Jun 2018 12:25:57 GMT

Hi HiroshiSatoh,

Reading the inputs.conf documentation it seems the "fschange" is deprecated since version 5.0
I will probably go with a UNIX script that works in that way:

Find all the files in folder A that:
are not older than 1 day
are closed
have a size lower than 10MB
based on the files found at point 1, generate symbolic links in folder B
Monitor with Splunk the symbolic links in folder B
Cancel symbolic links older than 2 days

Best Regards,
Edoardo

Re: Blacklist files greater than a certain size from inputs.conf

HiroshiSatoh — Tue, 26 Jun 2018 12:46:12 GMT

fschange has not been deleted yet. It is convenient so I am using it now.

Re: Blacklist files greater than a certain size from inputs.conf

adonio — Tue, 26 Jun 2018 13:44:29 GMT

i recommend not to use symbolic link, splunk is not always good in understanding it with the monitor stanza
better find files matching criteria - > copy to a new directory -> monitor that directory

Re: Blacklist files greater than a certain size from inputs.conf

edoardo_vicendo — Mon, 05 Nov 2018 17:24:24 GMT

Hi All,

At the end I went with a UNIX script that works in that way:

Find all the files in folder A that:

are not older than 5 minutes (see find with -mtime)
are closed
have a size lower than 16KB (that in my case means around 400 lines)

Those files will be then copied in the folder moniterd in batch mode by the forwarder.
For the files greater than 16KB I have done a head -200 and tail -200 and copied as well with the same original file name in the folder moniterd in batch mode by the forwarder.

Thanks to all for your suggestions!

Best Regards,
Edoardo