Blacklist files greater than a certain size from inputs.conf

[edoardo_vicendo]

Hi All,

I have to monitor a folder where there are very huge files with file name automatically generated.
Is there some way (instead of write a custom UNIX script that moves only small files to another folder that will be then monitored by the forwarder) to blacklist files that have a size greater than (suppose) 10 MB?

Any other suggestion with Splunk stanza attributes is appreciated.

Thanks a lot,
Edoardo

[edoardo_vicendo · Accepted solution]

Hi All,

At the end I went with a UNIX script that works in that way:

Find all the files in folder A that:

• are not older than 5 minutes (see find with -mtime)
• are closed
• have a size lower than 16KB (that in my case means around 400 lines)

Those files will be then copied in the folder moniterd in batch mode by the forwarder.
For the files greater than 16KB I have done a head -200 and tail -200 and copied as well with the same original file name in the folder moniterd in batch mode by the forwarder.

Thanks to all for your suggestions!

Best Regards,
Edoardo

[HiroshiSatoh]

It can not be executed because there is no size limit parameter in "monitor".
If it is "fschange", it may be restricted by "endEventMaxSize". However, it captures the entire file, not differential import.

[edoardo_vicendo]

Hi HiroshiSatoh,

Reading the inputs.conf documentation it seems the "fschange" is deprecated since version 5.0
I will probably go with a UNIX script that works in that way:

1. Find all the files in folder A that:
   are not older than 1 day
   are closed
   have a size lower than 10MB

2. based on the files found at point 1, generate symbolic links in folder B

3. Monitor with Splunk the symbolic links in folder B

4. Cancel symbolic links older than 2 days

Best Regards,
Edoardo

[adonio]

i recommend not to use symbolic link, splunk is not always good in understanding it with the monitor stanza
better find files matching criteria - > copy to a new directory -> monitor that directory

[HiroshiSatoh]

fschange has not been deleted yet. It is convenient so I am using it now.

[adonio]

hello there,
if the files have some naming convention you can follow, you can apply rules based on their name.

[edoardo_vicendo]

Hi Adonio,

Unfortunately there is no logic in the file name.

Thanks a lot,
Edoardo

[adonio]

@HiroshiSatoh has the right answer imho