We are thinking of using directory recursion to collect logs from various servers, but what's the overhead on the system?

[gopal20]

In a large enterprise with thousands of IIS, apache and tomcat servers, with each server having multiple web or app instances and with each instance having different log file locations, any suggestions on an efficient way to collect these logs?

We are thinking of using directory recursion, but what’s the overhead on the system?

[thomrs]

I gather lots of logs from our syslog server recursively and no issues. The host_regex in the inputs.conf helps keeps all the server names in order. If you use log rotation to move logs make sure you have a good whitelist/blacklist strategy.

We also had to up the bandwidth the UF uses to keep up with the amount of data in limits.conf

http://docs.splunk.com/Documentation/Splunk/6.2.2/admin/inputsconf

http://docs.splunk.com/Documentation/Splunk/6.2.2/Admin/Limitsconf

TO keep an eye on things take a look at the SOS and Deployment Monitor apps.

https://apps.splunk.com/app/748/

https://apps.splunk.com/app/1294/

[gopal20]

Using directory recursion generated 30% increase in CPU usage by Splunkd. Not a good option for our environment.