Archive

What is the limit of number of files a Universal forwarder can monitor?

dineshraj9
Builder

I have a server which has 1million+ files, but at a time 5k files being generated. Splunk installation is unable to work with such high volumes.

What would be the ideal approach to onboard such volume of logs from a server?

Tags (1)
1 Solution

HiroshiSatoh
Champion

Did you see this answer?

https://answers.splunk.com/answers/294295/is-there-a-limit-best-practice-to-how-many-data-in-1.html

In addition to the UF side, it is necessary to consider the configuration of the Splunk server and the daily log volume.

How about taking 1 UF, 1 pipeline(Default configuration) and trying out where the bottleneck will be?
However, please set the number of open files and transfer speed to appropriate values.

View solution in original post

ddrillic
Ultra Champion

ignoreOlderThan can be useful in your case at inputs.conf

0 Karma

gjanders
SplunkTrust
SplunkTrust

Do keep in mind what ignoreOlderThan does, if your file stops been written to for a period of time longer than the ignoreOlderThan setting, the forwarder will not attempt to read that file until next restart.

For example, you are monitoring /a/webserver.txt
You have an ignoreOlderThan = 2d setting
/a/webserver.txt stops updating for >2 days.

Even if /a/webserver.txt gets updated , the universal forwarder will never attempt to read it, unless of course your restart the universal forwarder in which case the 2 day rule would apply again.

0 Karma

HiroshiSatoh
Champion

Did you see this answer?

https://answers.splunk.com/answers/294295/is-there-a-limit-best-practice-to-how-many-data-in-1.html

In addition to the UF side, it is necessary to consider the configuration of the Splunk server and the daily log volume.

How about taking 1 UF, 1 pipeline(Default configuration) and trying out where the bottleneck will be?
However, please set the number of open files and transfer speed to appropriate values.

View solution in original post

dineshraj9
Builder

Thanks! Above post answers most of my queries 🙂

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!