Knowledge Management

Best Practice for Large Directory of Files

chrisboy68
Contributor

Hi,

I have some very large directorys. Here is my input.conf

[monitor://\\server\folder]
disabled = false
host = myhost
index = mylogs
sourcetype = mytasks
ignoreOlderThan = 2d
whitelist = (MYTasks\[EXPORT.*.log|MYTasks\[IMPORT.*.log)

When I check the number of files (Data inputs » Files & directories), I see 3840 files. Because of the Whitelist and modtime, most are ignored. Question, is this common? Seems inefficient for Splunk to monitor "skipped" files, I'm not sure if it has to re-read/touch these at every restart. I get a ton of files listed in "services/admin/inputstatus/TailingProcessor:FileStatus". If this is normal I'll move on, just trying to make sure my instance is performing optimally.

Thank you,

Chris

Tags (1)
0 Karma

woodcock
Esteemed Legend

If you cannot remove the files where they are, you are eventually going to have so many files for Splunk to dig through that the forwarder will be hopelessly slow in locating new events. The best way to handle it is to remove files that Splunk is not interested in or which Splunk has forwarded and will never update (check out the batch input type). If this cannot be done, then you can have Splunk monitor a different directory and you write a script and schedule it on a cron job so that it creates (and removed) soft links for the files that Splunk needs to watch.

0 Karma

chrisboy68
Contributor

Thank you for your reply. This made me dig more and I noticed that some directory's were being archived. Then with more reading I saw

recurse = [true|false]
* If true, recurse directories within the directory specified in [fschange].
* Defaults to true.

Hence, changing recurse=false, really reduced the i/o read time.

Thanks!

Chris

0 Karma

woodcock
Esteemed Legend

Yes, I would have suggested this if you had a said anything about subdirectories. It works very well.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...