Getting Data In

Is there a limit on the number of files a forwarder can monitor?

glitchcowboy
Path Finder

Is there a limit to the number of files an Universal Forwarder can monitor? When should I install multiple Universal Forwarder instances on a host?

We have several servers that will have about 200 log files to watch and each will be represented by a single [monitor] stanza in inputs.conf

Is there a performance bonus or problem with multiple forwarder instances?

1 Solution

gkanapathy
Splunk Employee
Splunk Employee

There is no hard limit, but there are practical ones depending on how many files are listed, how many of those are blacklisted, how many of those remaining are actively being written to, and how often. At the high end there is also question of how much latency in event indexing you are willing to tolerate. It also depends on how quickly your disk/filesystem can respond.

However, in the case of explicitly listing 200 files individually, that should be well within the ability of a forwarder to monitor with a second's delay. I would not worry unless you have on the order of 100,000 files listed and thousands being written to.

View solution in original post

glitchcowboy
Path Finder

Thanks (both of you) for your experience. I also had some feedback from support saying basically the same thing - there's no need for multiple forwarder instances and a single forwarder should do everything we need.

0 Karma

cwacha
Path Finder

We have one univeral forwarder running on a solaris system and it is monitoring around 30000 files where 1% of them is being constantly written to. Its working using the 4.3.3 code but we get delays of several minutes until events show up in the search GUI. If we only monitor one single file on this box the transport delay lies within seconds.

gkanapathy
Splunk Employee
Splunk Employee

There is no hard limit, but there are practical ones depending on how many files are listed, how many of those are blacklisted, how many of those remaining are actively being written to, and how often. At the high end there is also question of how much latency in event indexing you are willing to tolerate. It also depends on how quickly your disk/filesystem can respond.

However, in the case of explicitly listing 200 files individually, that should be well within the ability of a forwarder to monitor with a second's delay. I would not worry unless you have on the order of 100,000 files listed and thousands being written to.

Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...