topic Re: Is there a limit on the number of files a forwarder can monitor? in Getting Data In

Is there a limit on the number of files a forwarder can monitor?

glitchcowboy — Wed, 05 Sep 2012 18:58:25 GMT

Is there a limit to the number of files an Universal Forwarder can monitor? When should I install multiple Universal Forwarder instances on a host?

We have several servers that will have about 200 log files to watch and each will be represented by a single [monitor] stanza in inputs.conf

Is there a performance bonus or problem with multiple forwarder instances?

Re: Is there a limit on the number of files a forwarder can monitor?

gkanapathy — Wed, 05 Sep 2012 20:57:10 GMT

There is no hard limit, but there are practical ones depending on how many files are listed, how many of those are blacklisted, how many of those remaining are actively being written to, and how often. At the high end there is also question of how much latency in event indexing you are willing to tolerate. It also depends on how quickly your disk/filesystem can respond.

However, in the case of explicitly listing 200 files individually, that should be well within the ability of a forwarder to monitor with a second's delay. I would not worry unless you have on the order of 100,000 files listed and thousands being written to.

Re: Is there a limit on the number of files a forwarder can monitor?

cwacha — Thu, 06 Sep 2012 09:18:06 GMT

We have one univeral forwarder running on a solaris system and it is monitoring around 30000 files where 1% of them is being constantly written to. Its working using the 4.3.3 code but we get delays of several minutes until events show up in the search GUI. If we only monitor one single file on this box the transport delay lies within seconds.

Re: Is there a limit on the number of files a forwarder can monitor?

glitchcowboy — Thu, 06 Sep 2012 13:50:25 GMT

Thanks (both of you) for your experience. I also had some feedback from support saying basically the same thing - there's no need for multiple forwarder instances and a single forwarder should do everything we need.