Getting Data In

Is there a limit on the number of files a forwarder can monitor?

glitchcowboy
Path Finder

Is there a limit to the number of files an Universal Forwarder can monitor? When should I install multiple Universal Forwarder instances on a host?

We have several servers that will have about 200 log files to watch and each will be represented by a single [monitor] stanza in inputs.conf

Is there a performance bonus or problem with multiple forwarder instances?

1 Solution

gkanapathy
Splunk Employee
Splunk Employee

There is no hard limit, but there are practical ones depending on how many files are listed, how many of those are blacklisted, how many of those remaining are actively being written to, and how often. At the high end there is also question of how much latency in event indexing you are willing to tolerate. It also depends on how quickly your disk/filesystem can respond.

However, in the case of explicitly listing 200 files individually, that should be well within the ability of a forwarder to monitor with a second's delay. I would not worry unless you have on the order of 100,000 files listed and thousands being written to.

View solution in original post

glitchcowboy
Path Finder

Thanks (both of you) for your experience. I also had some feedback from support saying basically the same thing - there's no need for multiple forwarder instances and a single forwarder should do everything we need.

0 Karma

cwacha
Path Finder

We have one univeral forwarder running on a solaris system and it is monitoring around 30000 files where 1% of them is being constantly written to. Its working using the 4.3.3 code but we get delays of several minutes until events show up in the search GUI. If we only monitor one single file on this box the transport delay lies within seconds.

gkanapathy
Splunk Employee
Splunk Employee

There is no hard limit, but there are practical ones depending on how many files are listed, how many of those are blacklisted, how many of those remaining are actively being written to, and how often. At the high end there is also question of how much latency in event indexing you are willing to tolerate. It also depends on how quickly your disk/filesystem can respond.

However, in the case of explicitly listing 200 files individually, that should be well within the ability of a forwarder to monitor with a second's delay. I would not worry unless you have on the order of 100,000 files listed and thousands being written to.

Get Updates on the Splunk Community!

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...