Splunk Search

Blacklist files greater than a certain size from inputs.conf

edoardo_vicendo
Contributor

Hi All,

I have to monitor a folder where there are very huge files with file name automatically generated.
Is there some way (instead of write a custom UNIX script that moves only small files to another folder that will be then monitored by the forwarder) to blacklist files that have a size greater than (suppose) 10 MB?

Any other suggestion with Splunk stanza attributes is appreciated.

Thanks a lot,
Edoardo

0 Karma
1 Solution

edoardo_vicendo
Contributor

Hi All,

At the end I went with a UNIX script that works in that way:

Find all the files in folder A that:

  • are not older than 5 minutes (see find with -mtime)
  • are closed
  • have a size lower than 16KB (that in my case means around 400 lines)

Those files will be then copied in the folder moniterd in batch mode by the forwarder.
For the files greater than 16KB I have done a head -200 and tail -200 and copied as well with the same original file name in the folder moniterd in batch mode by the forwarder.

Thanks to all for your suggestions!

Best Regards,
Edoardo

View solution in original post

0 Karma

edoardo_vicendo
Contributor

Hi All,

At the end I went with a UNIX script that works in that way:

Find all the files in folder A that:

  • are not older than 5 minutes (see find with -mtime)
  • are closed
  • have a size lower than 16KB (that in my case means around 400 lines)

Those files will be then copied in the folder moniterd in batch mode by the forwarder.
For the files greater than 16KB I have done a head -200 and tail -200 and copied as well with the same original file name in the folder moniterd in batch mode by the forwarder.

Thanks to all for your suggestions!

Best Regards,
Edoardo

0 Karma

HiroshiSatoh
Champion

It can not be executed because there is no size limit parameter in "monitor".
If it is "fschange", it may be restricted by "endEventMaxSize". However, it captures the entire file, not differential import.

0 Karma

edoardo_vicendo
Contributor

Hi HiroshiSatoh,

Reading the inputs.conf documentation it seems the "fschange" is deprecated since version 5.0
I will probably go with a UNIX script that works in that way:

  1. Find all the files in folder A that:
    are not older than 1 day
    are closed
    have a size lower than 10MB

  2. based on the files found at point 1, generate symbolic links in folder B

  3. Monitor with Splunk the symbolic links in folder B

  4. Cancel symbolic links older than 2 days

Best Regards,
Edoardo

0 Karma

adonio
Ultra Champion

i recommend not to use symbolic link, splunk is not always good in understanding it with the monitor stanza
better find files matching criteria - > copy to a new directory -> monitor that directory

0 Karma

HiroshiSatoh
Champion

fschange has not been deleted yet. It is convenient so I am using it now.

0 Karma

adonio
Ultra Champion

hello there,
if the files have some naming convention you can follow, you can apply rules based on their name.

0 Karma

edoardo_vicendo
Contributor

Hi Adonio,

Unfortunately there is no logic in the file name.

Thanks a lot,
Edoardo

0 Karma

adonio
Ultra Champion

@HiroshiSatoh has the right answer imho

0 Karma
Get Updates on the Splunk Community!

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...