Is there a limit on the number of files a forwarder can monitor?

[glitchcowboy]

Is there a limit to the number of files an Universal Forwarder can monitor? When should I install multiple Universal Forwarder instances on a host?

We have several servers that will have about 200 log files to watch and each will be represented by a single [monitor] stanza in inputs.conf

Is there a performance bonus or problem with multiple forwarder instances?

[glitchcowboy]

Thanks (both of you) for your experience. I also had some feedback from support saying basically the same thing - there's no need for multiple forwarder instances and a single forwarder should do everything we need.

[cwacha]

We have one univeral forwarder running on a solaris system and it is monitoring around 30000 files where 1% of them is being constantly written to. Its working using the 4.3.3 code but we get delays of several minutes until events show up in the search GUI. If we only monitor one single file on this box the transport delay lies within seconds.

[gkanapathy · Accepted solution]

There is no hard limit, but there are practical ones depending on how many files are listed, how many of those are blacklisted, how many of those remaining are actively being written to, and how often. At the high end there is also question of how much latency in event indexing you are willing to tolerate. It also depends on how quickly your disk/filesystem can respond.

However, in the case of explicitly listing 200 files individually, that should be well within the ability of a forwarder to monitor with a second's delay. I would not worry unless you have on the order of 100,000 files listed and thousands being written to.