Getting Data In

We are thinking of using directory recursion to collect logs from various servers, but what's the overhead on the system?

gopal20
New Member

In a large enterprise with thousands of IIS, apache and tomcat servers, with each server having multiple web or app instances and with each instance having different log file locations, any suggestions on an efficient way to collect these logs?

We are thinking of using directory recursion, but what’s the overhead on the system?

0 Karma

thomrs
Communicator

I gather lots of logs from our syslog server recursively and no issues. The host_regex in the inputs.conf helps keeps all the server names in order. If you use log rotation to move logs make sure you have a good whitelist/blacklist strategy.

We also had to up the bandwidth the UF uses to keep up with the amount of data in limits.conf

http://docs.splunk.com/Documentation/Splunk/6.2.2/admin/inputsconf

http://docs.splunk.com/Documentation/Splunk/6.2.2/Admin/Limitsconf

TO keep an eye on things take a look at the SOS and Deployment Monitor apps.

https://apps.splunk.com/app/748/

https://apps.splunk.com/app/1294/

0 Karma

gopal20
New Member

Using directory recursion generated 30% increase in CPU usage by Splunkd. Not a good option for our environment.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud's AI Assistant in Action Series: Auditing Compliance and ...

This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...