Best Practice for Large Directory of Files

[chrisboy68]

Hi,

I have some very large directorys. Here is my input.conf

[monitor://\\server\folder]
disabled = false
host = myhost
index = mylogs
sourcetype = mytasks
ignoreOlderThan = 2d
whitelist = (MYTasks\[EXPORT.*.log|MYTasks\[IMPORT.*.log)

When I check the number of files (Data inputs » Files & directories), I see 3840 files. Because of the Whitelist and modtime, most are ignored. Question, is this common? Seems inefficient for Splunk to monitor "skipped" files, I'm not sure if it has to re-read/touch these at every restart. I get a ton of files listed in "services/admin/inputstatus/TailingProcessor:FileStatus". If this is normal I'll move on, just trying to make sure my instance is performing optimally.

Thank you,

Chris

[woodcock]

If you cannot remove the files where they are, you are eventually going to have so many files for Splunk to dig through that the forwarder will be hopelessly slow in locating new events. The best way to handle it is to remove files that Splunk is not interested in or which Splunk has forwarded and will never update (check out the batch input type). If this cannot be done, then you can have Splunk monitor a different directory and you write a script and schedule it on a cron job so that it creates (and removed) soft links for the files that Splunk needs to watch.

[chrisboy68]

Thank you for your reply. This made me dig more and I noticed that some directory's were being archived. Then with more reading I saw

recurse = [true|false]
* If true, recurse directories within the directory specified in [fschange].
* Defaults to true.

Hence, changing recurse=false, really reduced the i/o read time.

Thanks!

Chris

[woodcock]

Yes, I would have suggested this if you had a said anything about subdirectories. It works very well.